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ABSTRACT 



The purpose of this research study was to undertake a systematic 
investigation into the relationships among: (1) the techniques used by 
search analysts during preliminary interviews with users before engaging 
in online retrieval of bibliographic citations; (2) the amount of new 
information gained by the user as a result of the search; and (3) the 
user's ultimate satisfaction with the quality of the items retrieved. 
A series of controlled experiments were conducted to explore the effects 
of two interview techniques: the conscious use of "open" and "closed" 
questions and the use of pauses of different lengths by the searph 
analyst during the online negotiation interview. Data. were collected^ 
on various aspects of the user's need for .information, the value he/she 
placed upon new knowledge, and the consequences of inadequate information. 
Analytical techniques included two-way analysis of variance and path 
analysis. While search analysts displayed no difficulty in asking 
open and closed questions, they found considerable difficulty in con- 
trolling the lengths of pauses. Among the findings were the following: • 
the asking of open and closed questions had a modest effect on the 
amount learned by the users; the type of pause did have a significant 
effect on the amount clients learned; average users satisfaction was 
higher when open questions were asked; overall satisfaction was lower 
when moderate pauses were used; those learning most about their topic 
were, overall, more satisfied than those who learned less; those placing 
high importance on the information obtained tended to have lower 
satisfaction scores. 
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PREFACE 

The research described in this report took place over a three 
year period from September 1979 through October 1982. The activities 
that occurred during that time took place approximately as follows. 
During the first year staff were hired, instruments were constructed, 
pre-tested and revised, analysts were trained, control data collected, 
and experimental treatments embarked upon in the field settings. The 
administration and completion of the open-closed questions and no pause- 
moderate pause experiments took place over an eighteen month period. 
Soon after the beginning of the experiments, data collection and coding 
began and continued until the end of the second year. The third year 
saw the analysis of the data and the preparation and writing of this 
final report. 

A word of explanation about the organization of the report is 
in order. During the course of the study much developmental work was 
undertaken to ensure that the items and instruments used would in fact 
measure what we wished them to. Their reliability and validity were 
assessed and methods were devised to analyze taped negotiation interviews. 
Descriptions of the procedures we developed, preliminary findings and 
implications were reported to the professional community at the annual 
meetings of the Canadian Association for Information Science, 1981 and 
1982. While these papers are- developmental , they nevertheless contain 
much that is valuable in a field where there is a paucity of hard data. 
Therefore, we felt that they should be included as part of our final 
report. They constitute Parts III and IV. 
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While on the one hand our work breaks new ground in the approach 
it takes and the analytical procedures developed, it nevertheless treats 
an area— the negotiation interview—that is well grounded in a rich 
scholarly tradition, that of the interview in traditional reference work. 
A substantial body of this literature was reviewed as well as that treating 
computer-based reference services. We felt that a thorough review of 
both these areas was necessary to enable us to understand the present 
state of knowledge in the field and thereby attempt to extend it. This 
substantial review of the literature constitutes Part II of this report. 

Finally, the main thrust of the research -is described in Part I. 
Each Part, though self-contained and complete in itself, contributes to 
form a comprehensive picture of our work over the past three years. 
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INTERVIEW TECHNIQUE AND USER SATISFACTION 
WITH ONLINE BIBLIOGRAPHIC RETRIEVAL 

In this part of the report are described the results of two field 
experiments that were conducted to determine the impact of different Interview 
techniques on the satisfaction of users with the results of on-line 
bibliographic searches. Each experiment involved two search analysts 
(specially trained reference librarians), each located in a special 
library serving a faculty of education. The first experiment involved 
asking different types of questions ("open" and "closed"); the second 
involved the use of pauses of different lengths ("no 11 pause and "moderate" 
pause). In addition to the experimental and criterion or output variables, 
several other variables — in particular, the amount learned as the result 
of conducting an online bibliographic search— were measured in order to 
understand the antecendents of user satisfaction. A framework drawn 
from the field of the economics of information was used to guide the 
selection of these variables. 

Background 

The topic Of the present study was suggested by the conclusions 
of a previous study by the authors (Lawton, Auster and To, 1979), in which 
we developed a systems model to be used in the evaluation and analysis 
of online bibliographic search services. The model, which was applied 
to the Educational Information Systems for Ontario (EISO), involved 
nine variables: method ^of contact (in person or phone), system used 
(SDC or Lockheed), turnaround time, connect time, strategy time, 
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citations printed 4 , price, amount learned by the user , and user 
satisfaction. The last variable, satisfaction, had five subscales: 
satisfaction with publicity material and directions, with the 
convenience of the service and helpfulness of the search analyst, with 
the timeliness of the service, with the quality of the technology, and 
with value of the bibliography and materials. 

Path analysis was used to analyze the evaluation data. This 
approach allows one to test the validity of causal inferences for pairs 
of variables while controlling for the effects of other variables (Nie, 
et al . 1975; Lawton, et al . 1979). 

In conclusion, we noted first that only one variable had a 

consistent effect on satisfaction, namely price. We cautioned, 

In interpreting this differential effect of 
price, it must be... [noted] that 9% of all users 
received free searches and most others paid 
$30 or more. It would appear then that when 
individuals are paying for bib! iographic search 
[sic], they expect more in the way of service 
than they would if they were receiving it for 
free. Yet, it is important 'to note that their 
expectations with regard to service do not carry 
over to their assessment of the value of the 
goods , which they apparently assess independently 
of amount paid (p. 39). 

A second variable, the amount learned about the topic searched, 

was also related to various satisfaction (subscale) scores with some 

consistency: 

This variable is an output variable in its own 
right, but intervenes between satisfaction and 
all process variables except price. It is 
positively related to overall satisfaction, 
satisfaction with time! iness of service, and, 
most important, satisfaction with the bibliography 
and materials. .. [This] highlights the idea that 
the client is motivated to learn and that if this 
learning does not occur dissatisfaction will result 
(p.39). 



Though the amount a user learned about his or her topic 

proved to be an important intervening variable in explaining user 

satisfaction, none of the process variables—method of contact system 

used, strategy time, connect time, turnaround time, or number Qf citations 

printed— he! ped to account for the* learning that took place. In 

conclusion, we noted: 

It would appear that explanations [of the amount 
learned] must be sought in the personality or 
knowledge of the client, the interaction between 
the client and search analyst, the quality of the 
search strategy, or the topic being searched, 
(p. 39) 

A second, exploratory, study refined our understanding of the 
interaction which took place during the search interview (Auster and 
Lawton, 1979). Findings suggested a model of the process with three 
component strands— the behaviour of the client, the questioning techniques 
of the search analyst, and the topic of the search itself. The search 
analyst's methods Included her use of empathy and non-verbal expression, 
open and closed questions, and a loose structuring of the interview into 
five stages. On the user's part, it appeared that his or her stock of 
knowledge, ability to express the content and type of information needed, 
and the value placed upon this information were the important variables. 

Interview Techniques 

The purpose of the interaction between the search analyst and the 
user is for the former, with his or her knowledge of ratrieval systems 
and data bases, to elicit information from the latter that can be 
translated into adequate search strategies for querying the data bases 
via search systems available (Atherton and Christian, 1977). The 
exploratory study suggested a number of factors that might explain the 



"success" of a given negotiation process. The literature on interview 
techniques suggested many others that might also be considered (Richardson, 
1965). Some of these appear to be more important than others, and some 
more manipulable than others. 

Two specific techniques that appeared salient in both the 
literature and the exploratory research were the type of question used 
and the "pace" of the interview. 

A fundamental distinction in types of question is that of open 
versus closed questions. Open questions are those which are general in 
nature and that do not suggest a specific category of response. Closed 
questions, in contrast, require only one or several words in response, 
usually of a categorical nature—a name, a number, yes or no, and so on. 
For example, "What is, your educational background?" is an open question; 
"What degrees do you hold?" is a closed question. Sometimes, phrasing may 
vary, however, and judgement is necessary; e.g., "Would you tell me about 
your educational background?" is clearly "open" in intent, though it can, 

■9 

technically, be answered by a yes or no. 

The literature on interview techniques suggests that open questions 
elicit longer responses that reveal more information about the interviewee's 
understanding—or "map"— of the world, than do closed questions. The 
latter tend to demand that the interviewees be fit into the categories 
present in the interviewer's mind. Responses tend to be short, though interviewees 
may take it upon themselves to explain their answers and thereby avoid 
the categories implicit in the question. 

-The "pace" of the interview, though not identified specifically 
in our research, was a factor suggested by the tendency we noted for the 
search analyst to dominate the direction of the interview, sometimes cutting 
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off the interviewee in order to proceed to the next item on his or her "agenda" 
The literature suggests that the ability of the interviewer to pace the 
interview— by allowing pauses of different lengths to occur—can affect the 
content of the interviewee's responses. In particular, allowing no pauses or 
extremely long pauses (over 10 seconds) tends to reduce the responsiveness 
of interviewees. The first because they cannot get a word in edge wise (the 
phenomena we perceived), and the second because long pauses apparently 
suggest to the interviewees a termination of interest by the interviewer 
and not just a pause in expectation of a response. 

Both of these factors — types of questions used and use of pauses- 
appeared both important and trainable; i.e., a search analyst could, it 
appeared, be coached in the techniques. Other techniques in the literature or 
that we had noted— e.g., empathy and non-verbal communication— seemed more 
difficult to affect, more difficult to measure, and possibly of less importance 

Economics of Information 

The notion that information has value suggests that the perspectives 
represented by the economics of information (Levin, 1980; Stiegler, 1961) 
and decision theory (Edwards, 1963; Sudman, 1976) may help in explaining a 
user's satisfaction with the results of a bibliographic search. As Levin 
notes, the key idea behind these frameworks is that information has both a 
cost and a benefit: 



The benefit that is attributable to information derives 
from its value in improving decision making and resul- 
tant outcomes. For example, the consumer who finds in 
an advertisement that he can purchase an item that he 
needs at a reduced price will receive benefit from that 
information that is equal to the price reduction. The 
cost of information refers to the resources required to 
collect, analyze, and disseminate it as well as the-cost 
.to the user of acting on it. Such costs include not onTy 
pecuniary ones that we can find on accounting statements, 
but also such "non-accounting" costs as the information 
user's time. . .(p. 7) 
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Levin provides a straightforward example of a person seeking a 

new car who visits a local automobile agency to ascertain prices. 

He selects a particular model that will satisfy 
his needs, but he decides to think about it 
before buying it. Outside of the automobile 
agency, he purchases a newspaper and turns to 
the auto section. There he finds that he can 
obtain the same model for $200 less, but the 
agency is 10 miles away. He returns to the 
automobile agency and shows them the advertisement, 
and they agree to reduce the price by $200 to make 
the sale. For the relatively nominal cost of 
the newspaper and the value of the man's time, he 
was able to receive a benefit of $200. 

Sudman (1976) indicates two factors that affect the value of 

information: (1) the degree of uncertainty, and (2) the nearness of 

the decision maker's prior expectation to the decision point. 

If his prior expectation is far higher or lower 
than the break-even point, there will be little 
or no value to the information, even if there is 
a high degree of uncertainty [and hence a great 
need for information]. On the otherhand, if the 
prior expectation is near the break-even point, 
information may be very valuable even with little 
uncertainty, (p. 93) 

In the case of Levin's automobile purchaser, we must infer that 
the initial price quotation was higher than the shopper's prior expectation; 
otherwise he would have purchased the car. Had the expectation been much 
lower—several thousand dollars--the additional information would have been 
worthless. As it was, a single piece of information was of considerable 
value, suggesting that he was very near to his break-even point in 
assessing the worth of the car to him as opposed to the other uses to 
which the money might be put. 

It would appear that this same logic would apply in the case of 
at least some users of a bibliographic search service. For example, 
a user who is a professional educator undertaking an evaluation of a 
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language arts program could face a decision concerning the creation of 
a new evaluation instrument, at considerable expense, or the adoption of 
an existing one. Information about similiar evaluation projects could * 
provide information that would reduce the educator's uncertainty, and 
hence aid in making a decision. 

There are other approaches to the issue of assessing the worth 
of information, however. Woodward (1980) argues that much information 
is used solely for cognitive purposes— learning new ideas, reconstructing 
old ones, etc. In this case, the information has no economic value. 
Still, it can be assumed that the individual has some prior knowledge, 
is seeking additional information, and that the information is important 
to the user who places some valuation upon it. 

The Problem 

This study, then, places the search for information within the 
context of the economics of information. The search interview is looked 
upon as a controllable process within the overall process that comprises the 
online bibliographic retrieval cycle—from the user's initial contact 
with a search service to his or her final assessment of the bibliography 
and materials obtained as a result of the search. An intermediate product 
of the search process, the amount a client learns about his or her topic, 
is viewed as a key Intervening variable which, in the end, will account 
for much of the variation in satisfaction. 

Two specific questions are asked: 

1. What are the relationships among various interview 
techniques (type of questions and length of pauses), 
the amount learned by the user, and the user's 
satisfaction? 
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2. What are the relationships among the user's 

prior knowledge of the topic, amount of information 
gained, assessment of the importance of the 
information sought, and satisfaction with the 
search results? 

The first question arises from the analysis of the interview 
process and the review of literature on the topic. Implicit within 
the question is a chain of relationships linking interview technique 
to the amount learned, and the latter with user satisfaction. In 
suggesting these linkages, four assumptions are made: 

1. that the user's responses to the search analyst's 
questions are related to the type of questions 
asked; 

2. that the content of the search strategies 

developed by the analyst are related to these responses; 

3. that the citations retrieved are closely related in 
content to the search strategy used; and 

4. that the information gained by the user is related 
to the content of the citations received. 

Evidence will be cited that supports the validity of each of 
these assumptions. 

The second question is suggested by the economics of information 
perspective and the factors suggested by Sudman— uncertainty and nearness 
of prior expectations to the decision point. In particular, it is assumed 
that the extent of an individual's prior knowledge reflects uncertainty, 
that the amount of information gained reflects the reduction in uncertainty 
and that a post facto assessment of the importance of the information 
retrieved or missed indicates how close the individual is (or was) to the 
decision point. 

Implicit in the question is the proposition that the greater the 
reduction in a user's uncertainty (i.e., gain in knowledge) and the 
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nearer he or she was to the decision point (i.e., the greater the 
importance placed on the information retrieved), the greater will be 
the person's satisfaction. Satisfaction, therefore, would probably 
be greatest when both the amount learned and the importance of the 
information were both high, and lowest when both of these factors 
were low. In the two intermediate cases— a little information that 
i* of high value, or an extensive amount of information that is of 
little value— one would expect middle levels of satisfaction. And of 
these two, one would expect higher satisfaction in the first case 
since the information was important; i.e., it appears that being 
near the break-even point is more important in explaining satisfaction 
than reducing uncertainty. As in the case of the car purchaser 
previously described, a little information was of considerable value 
because the purchaser was near the decision point. 

If the preceding analysis is correct, then it is possible to 
hypothesize that satisfaction with the results of a bibliographic 
search would be ranked in the order suggested in Figure 1. 



Amount Learned 



High 



Low 



High 



1 



2 



Importance 



Low 



3 



4 



Figure 1. Hypothesized rank ordering of satisfaction 

scores in relation to the amount and importance 
of information obtained. 
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If one were to emphasize the cognitive contribution of information 
instead of its contribution to decision making, the same two variables, 
amount learned and its importance, would appear relevant, though the 
second is less so since no "decision point" can be said to exist. Irk 
this case, the ranking of the satisfaction scores for the two middle \ 
categories would be reversed, since the amount learned would be more 
dominant than the importance of the material per se . 

In a sense, then, though the rationale for this study is based . 
on the economics of information, an alternative perspective is also being 
tested. 

Methodology 

The core of this study are field experiments in which different 
types of interview techniques were used by two search analysts, with the 
(inferred) effects of the search results on the users being assessed 
by measures of both the amount users learned and their satisfaction. 
In supplementary analyses, the effects of non-experimental independent 
variables, such as a user's prior -knowledge and the importance of 
information sought, are assessed. As well, the latter variables are 
used as control variables in a path analysis that provides an integrated 
explanation of user satisfaction within the framework of the economics 
of Information. 

The first of the two interview techniques used in the experiment 
involved the use of either "open" or "closed" questions by the interviewer. 
The second involved the use of either "no" or "moderate" pauses. Both of 
these techniques alter the normal interaction that takes place by the 
interviewer (the search analyst) and the interviewee (the user.). 

* . 1J 
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The design of the first, experimental phase of the study can be best 
expressed as being composed of two, two-way analyses of variance with post- 
tests only. The experimental variables were question type and length of 
pause. Each had three levels— control , open, and closed questions, and 
control, no pause and moderate pause. Separate experiments were conducted 
because of the difficulty of training people to do two things at once; i.e., 
varying the type of questions asked and length of pauses. The same control 
group served for both experiments, however. 

The second independent variable was "analyst"; i.e., the first or 
second search analyst. This variable was not introduced in order to 
discover which analyst was superior, but to control for the different 
skills of the analyst and other "nuisance" variables correlated with 
analyst, such as the type of user served, indiosyncracies of search 
technique, and so on. 

The dependent variables consisted of the amount learned (a variable 
that,- in the overall framework of the study, is considered an intervening 
variable) and several satisfaction variables—satisfaction with the length 
of the bibliography, the value of the bibliography, the value of the materials 
located via the bibliography, the currency of the search results, and the 
utility of the search results for the intended purpose. 

Neither the users nor the analysts were selected at random. 
The former were screened only to ensure that they had not been included 
in the study before and were conducting retrospective searches. The 
latter were two analysts known to the researchers who were willing to 
participate. Both were employed in libraries in faculties of education 
in Canadian universities. One of the faculties involved a large number 
of masters and doctoral students, while the other served a small number 
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of graduate students at the masters level and a large number of 
teachers-in-training. The first analyst was female and was new to her 
position. The second was male and had several year's experience at 
on! ine searching. 

The search analysts each conducted a total of 75 interviews— 
15 control interviews, 15 with open questions, 15 with closed questions, 
15 with no pause, and 15 with moderate pauses. The control or "naive" 
interviews were conducted first, before the search analysts were trained 
in interview techniques (Appendix A). Subsequently, they interviewed users 
administering the four techniques in random order. To tell them which type 
of technique to administer for each search, a 1 ist numbered 16 to 75 was 
prepared (the first 15 interviews being the "control" group); opposite 
each number was an indication as to whether the search was to use open 
questions, closed questions, no pauses or moderate pauses. The latter 
four options were arranged randomly by use of a random number table. 
Digits 1, 2, 3, and 4 were assigned to the four options and the order 
of the four techniques was determined by the order of these digits in 
the table. Other digits were ignored, and each technique was listed 
only 15 times (Appendices B and C). 

At the time of the interview, a tape recording was made and the 
search analyst completed "Search Analyst's Record of Online Search" 
(Appendix D). At that time or later, depending on the situation, a 
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cbmplete search strategy was developed, recorded, and used in completing 
the search. A hard copy record of the actual search was also retained. 

Two to four weeks after a search had been completed, the user 
was telephoned and an evaluation interview was conducted, the results of 
which were recorded on a form "User Response to Online Search Request 1 ' . 
In cases where the user could not be contacted by phone, a copy of 
the form was mailed with a stamped return envelope enclosed (Appendix E). 

Subsequent to the collection of data, recordings of interviews were 
played in order to determine if the treatment had been administered; that 
is, if the search analysts had indeed used different interview techniques. 
The proportion of open and closed questions actually asked were recorded, 
as were the lengths of pauses (Appendix F) . 

Records of search strategies were also examined and a number of 
characteristics related to the complexity of the strategy were recorded. 

Finally, all data were keyed into a data file for computer 
analysis using the Statistical Package for the Social Sciences (SPSS). The 
initial analysis used two-way ANOVA, as planned. 

The second research question was also answered using a two-way 
analysis of variance, as is suggested by Figure 1. In particular, the 
users' responses to questions as to the importance and amount of 
information gained were dichotomized into high. and low, and two-way ANOVA 
was carried out to assess the variables' effects on satisfaction and to 
determine the rank ordering of the cell means. 

Finally, a path model was developed that integrated both the 
experimental variables (question type and pause) and those related to the 
economics of information framework (see Figure 3). Two path analyses 
were necessary since the data set for the "question" experiment was 
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separate from that for the "pause" experiment. The control group was 
dropped for this analysis so that question and pause would be dichotomous 
variables appropriate to path analysis. In addition, a variable measuring 
prior knowledge was added as a control variable. In theory, it should 
have been used in the preceding analysis, but could not be due to limita- 
tions in the cell sizes. Hence, this path analysis may be the best 
overall test of the framework suggested by the economics of information. 

Measurement of Variables 

The six variables used in the various of analyses are listed in 
Table 1. The first, third, and fourth (ANALYST, PREKNOWL, IMPORT) are 
independent variables; variables 2A and 2B (QUEST and PAUSE) are the 
experimental variables; variable 5 (NEW KNOWL) is an intervening variable 
used as both an independent and dependent variable and variable 6 (SATISFAJ) 
is used as the primary dependent variable. 

Each of variables three through six is measured by two or more 
items (Appendix G ). 

PREKNOWL, a measure of the extent of a user's prior knowledge 
about the search topic, has six items and ranges from a minimum of 6 to 
a maximum of 15. The mean response for the 117 respondents wa*s 12.2 with 
a standard deviation of 2.05. The Hoyt estimate of reliability for the 
scale is 0.67. 

IMPORT, a measure of the importance of the information sought in 
a search, has eight items. The minimum score is 6 and the maximum is 
30. The mean for the 103 respondents for whom data were available was 
11.9 with a standard deviation of 3.97. The Hoyt estimate of 
reliability is 0.69. 



Since this is a key variable, assessing how close the 
respondent is to his or her break-even point in making a decision, 
it is important to note the content of the items. Six items ask how 
missing relevant citations would have affected the purpose for which 
the search was placed* , and the user's financial status. Two items 
seek information on the total number of people that missed information 
might affect. We believe this scale to be a 

reasonably valid measure of nearness to the decision point. If missing 
50% of the available information was of no consequence, then either the 
decision or project could not be very much in question. At the same 
time, these questions, by considering the financial and social impact 
of the information sought, probably tap a somewhat broader definition 
of the importance of information than is required. 

NEW KNOWL, with just two items, is a measure of the amount an 
individual learned about his or her topic as the result of a search. 
In part, this learning is projected rather than completed, since the 
second item concerns the amount of material the user planned to 
read. The scales 1 minimum and maximum are 1 and 9, respectively; the 
112 respondent's average score was 5.13 with a standard deviation of 
2.06. The" Hoyt estimate of reliability is .65. This scale was 
reduced to the best two items from the five originally asked users. 

Satisfaction has five items which are treated separately as well 
as totalled as SATISFAC, the overall satisfaction score. The minimum 
possible score is 5 and the maximum 25. The 101 respondents had an 
average of 18.3 with a standard deviation of 3.83. The mean translates 
into an average of 3.6 on a 5-point scale with 1= very low and 5=very 
high. The Hoyt estimate of reliability for the scale is 0.86. 



Findings 

This section is divided into five parts. The first provides, 
an overview of the characteristics of the users of the two bibliographic 
search services and of their searches; i.e., their length, cost, etc. 
The second section includes evidence that treatments were given and 
that the four assumptions required to (^nswer , the first research question 
are valid. In the third section, the first research question is answered 
and in the fourth, the second question. In the final section, the path 
analyses that integrate the two questions are presented. 

Characteristics of Users 

Summary statistics for the users served by the two search 
analysts are given in Table 2. Both served users drawn from a 
university community, and both served about equal numbers of men and 
women. The users of the first search analyst, however, tended to 
be older than those of the second and were more likely to hold a masters 
or doctoral degree or be enrolled in a doctoral program. 

The purposes users reported for conducting searches were rather 
similar. Assignments were the most common reason (Table 3). The purposes 
reported by the users of the second analyst were somewhat more varied than 
were those reported by users of the first analyst. 

In both cases, essentially all users made contact in person. 
It is probable that both services had substantial numbers of users who 
phoned in or wrote. The requirement that the search interview be 
recorded effectively eliminated such users from the study. 

Most users supplied the search analyst with synonyms for use 
in the search strategy. Only the second analyst reported substantial 
numbers who assisted with the logic. 
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While the tfctal time spent doing a sincpe search, about 42 
minutes, was the same for both analysts, this time was spent differently. 
The firsu analyst conducted an extensive interview, then developed a 
strategy, and ran the search. The second began with a short interview, 
then developed the strategy and went online while interacting with the 
user While the connect time averaged slightly longer for the second 
analyst, it should be noted that this analyst often" printed some citations 
online while the first analyst rarely did so. 

The two most commonly used search systems were those offered by 
Lockheed Information Service and by BRS. SDC and QL (Quick Law) were 
never used. 

The two analysts tended to search rather different data bases. 
ERIC was used in over 90% of all searches by the first analyst, but in only 
37% of all searches by the second. The latter tended to favour 
Psychological Abstracts, the Social Science Citation Index (SSCI) and 
Dissertation Abstracts. 

The lengths of the bibliographies produced by the two search 
analysts were quite similar. However, virtually all of the first analyst's 
bibliographic citations were printed offline and mailed to the user, 
whereas a third of the second analyst's citations were printed online 
and given to the user at the time of the search. The second analyst also 
used the online citations to verify with the user that the search was 
on target. 

Prices averaged much higher for the second search analyst's users 
than for the first's. A majority of the latter received free 

searches. 

A major difference in style is evident in the two analysts' 
involvement with the user, during a search. As suggested above, the 



first analyst rarely had users present when doing a search, and hence 
rarely interacted with them. The second analyst almost always did the 
search while they were present, and usually interacted with them. 

Overall, then, it appears that two rather different settings 
were used for this study. At the first, the users tended to be older 
and more experienced, while the search analyst was new and used rather 
restricted methods of operation. Rarely were users present when the 
search strategy was worked out and the search conducted. At the second 
location, the clientele tended to be younger, but the analyst was more 
experienced and showed considerable variation in the choice of systems and 
data bases. Finally, users at the first location rarely paid for 
their searches, while those at the second location usually did so. 

Treatments and Assumptions 

The two search analysts were taught the basics of interview in 
a one day training session (Appendix A ) which emphasized the asking 
of open and closed questions, and the use of pauses. The analysts were 
to use one of the techniques in each of the interviews conducted ^as 
part of the study. 

One ought not assume, however, that saying and doing are one 
and the same. Hence, it was deemed necessary to verify that the various 
techniques were actually used when they were supposed to be. In 
addition, it was clear that verification, to the extent possible, of 
the assumptions made in linking interview techniques to user satis- 
faction was desirable. 
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Table 4 presents data verifying that the experimental treatments 
were, in fact, administered. In the first part of the table, the propor- 
tion of all questions asked by the search analysts that were open is 
treated as the dependent variable, while the three levels of the 
experimental variable (control, open and closed) are treated as the 
independent variable. The data clearly show that, on average, twice as 
many open questions were asked in the "open" interviews as in the 
"closed" interviews. The us^ of open questions in the "control" 
interviews was more common than in the closed interviews, but less 
common than in the open interviews. The relatively low frequency of 
open questions in all interviews suggests that it is difficult to frame 
f such questions while engaged in negotiating a search. 

Pauses of different lengths were also administered, though the 
average pauses, which were to be of 10 seconds in the moderate pause 
category, in fact only averaged a bit over two seconds. Pauses in the 
"no pause" category of search interviews averaged 0.6 seconds. Pauses 
during control searches were essentially non-existent-conversation 
was continuous. Measurements were not made for the pauses in these 

interviews, however. 

It is doubtful that the treatment administered during interviews 
in which pauses were to be used conformed to that planned. Both 
conversations with the analysts and close monitoring of the tapes 
revealed the analysts found silences very awkward. Often, they would 
turn to writing or looking in a thesaurus to avoid tension while trying 
to prolong pauses. 
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The process used to map out interviews in order to measure the 

proportion of open questions and lengths of pauses is rather complex and 

is treated in detail elsewhere (Auster, Lawton and Currie, 1982). (See Part III 
of this report.) 

Four assumptions concerning the search process were stated 

earlier; ideally, each should be validated. 

To verify the first (that a user's responses to the search 
analyst's questions are related to the types of questions asked) 25 
open and 25 closed questions were randomly selected from 50 tapes 
which were themselves selected at random. For each of these 100 
questions (50 per analyst), the number of words in the response given, 
the number of seconds in the response, and the number of major concepts 
in the response were recorded. If responses having more words and 
concepts, and being of longer duration, followed open as opposed to 
closed questions, then the assumption would be validated for type of 
question asked. 6 

A similar approach was taken to validate the effects of pauses. 
%For this the ten most extreme pauses for each analyst were selected; 
that is, the five longest and five shortest pauses were chosen for each 
analyst, the long pauses from "pause" interviews and the short pauses from 
"no pause" interviews. The average length of the five shortest pauses was 
0.0 seconds (i.e., no pause) and the average length of the five longest pauses 
was 23.0 seconds. After the identification of each pause (or "turnover" 
in conversation when there was no pause), the number of words, seconds 
and concepts in the response that followed were recorded. 

Table 5 reports the data required to validate the fact that 
questions and pauses matter. In the case of open and closed questions, 
the relationship to the characteristics of responses is as expected: 
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responses to open questions tended to have more words and concepts, and 
take longer to complete, than did those to closed questions. It ought 
be noted, however, that there was considerable variation from question 
to question, as revealed by the large standard deviations. 

The responses after pauses, however, were not longer, in words, 
seconds or concepts, than on occasions when there was no pause. Indeed, 
just the opposite is true. It may be that the excessive length of the 
pauses included in this sample interrupted the flow of conversation, 
created tension, and muted responses. Perhaps further investigation into 
the effects of moderate pauses (1-10 seconds) would have confirmed our 
expectations. However, the findings in Table 5 are consistent with 
results reported later, suggesting that the long pauses that occurred 
during these interviews were more important to the effectiveness of 
interviews than any moderate pauses that may have occurred. Also, the 
extremely large standard deviations in the numbers of words and seconds 
in responses to situations in which no pause occurred suggests that 
generalizations are risky. 

To verify tne second assumption—that the search strategy 
developed by the analyst would be closely related to the responses given 
by the user --ten case studies were carried out in which the vocabulary 
used by the user and the discrete terms and concepts employed by the 
analyst were compared. In practice, five interview tapes were selected 
for each analyst and all key terms and concepts employed by the user in 
his or her description of the topic being searched were recorded. The 
two lists were then compared. 

In all cases studied, there was an extremely close correspondence 
between the vocabularies employed by the users in the interview and by the 
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search analysts in the strategies they prepared. For example, one user 
indicated an interest in research on the administrative role of secondary 
school principals. The user wanted studies of actual activities rather 
than opinions as to what principals should do. The search analyst developed 
three main conceptual groups: administration (principal, assistant 
principal, school administration, school personnel, chief administrators , 
high school supervisors); research (research, use studies, case studies, 
surveys, measurement, questionnaires, research tools, interviews); and 
secondary education (secondary grades, secondary schools, senior high 
schools, high schools, junior high schools). 

In general, the search analysts used more terms, than did the 
user though this phenomenon was more apparent with the first analyst 
then the second. 

It would be wrong to assume, however, that the interview process 
described here is considered simply a matter of "twenty-one questions 1 '. 
Often, the client had only a vague idea of the information he or she 
desired, and the interview became very much a negotiation process in which 
the content and bounds of the bibliography were decided (see Auster and 
Lawton, 1979). 

Sometimes, it appears that the negotiated settlement did not 
prove satisfactory to the user. For example, one user interested in 
the career expectancies of minority group members was dissatisfied with a 
search whose key terms included "ethnic group" "career aspirations" 
"career planning," and "career interests." In the evaluation interview, 
the user indicated that she had found the material she wanted by conducting 
a manual search. The "actual" topic dealt with the percentage of minorities 
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actually represented in various occupations. "Expectancy" reflected not 
a psychological construct, but a statistical average; this distinction 
had not emerged during the search interview. Perhaps the user, in her 
own mind, had not yet made the distinction. 

A second approach to validating this assumption is to relate the 
length of the entire interview and search process to the complexity of 
the search strategy, in the belief that more complex search strategies 
would result from searches that required more interview, strategy and 
connect-time. 

Complexity of search strategy was measured by a five-part index 
consisting of the sum of the following items: (1) the number of key 
terms used in the search strategy, (2) the number of operations (e.g., 
"and," "or" and "not" employed to join terms together into concepts and 
to restrict major concepts to specific areas), (3) the total number of 
mr.jor concepts in the strategy, (4) the total number of steps in the 
online search strategy, and (5) the total number of data bases finally 
searched. The total length of the search process was measured by summing 
the interview time, strategy time, and connect time. 

The correlation between the length of the search process and 
complexity of search strategy was 0.43 (pZ.001), indicating that there 
was a reasonably strong link between the two. 

Assumption three stated that the citations retrieved would be 
closely related to the content of the particular search strategy used. 
As a rule, one might expect more complex search strategies to produce 
bibliographies with a higher number and percentage of hits than would 
less complex ones. Since we could not control for subject matter searched, 
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and since only one strategy was used to conduct the search for any given 
request, we concluded that it was impossible to satisfactorily validate 
this item within the design of the study. To do so would require using 
several different strategies to search the same topic, followed by the 
determination of the number and percentage of hits for each strategy. 

To fully validate the final assumption, that the amount of 
information gained is related to the content of the citations received, 
would require development of a series of subject matter tests on different 
topics to be administered before and after searches. This was not 
practicable in a field setting. However, indirect validation of this 
assumption was provided in an earlier study (Lawton, Auster and To, 1979) 
wherein the correlation between the amount a user learned about his or 
her topic was found to have a 0.67 correlation with the user's satis- 
faction with the value of the bibliography and materials. 

Interview Techniques 

Results of the experiments to answer the first research question 
concerning the effects of types of questions and lengths of pauses on 
the amount a user learns as a result of a search and his or her overall 
satisfaction are presented in tables 6 and 7. Two-way analysis of 
variance - was used, with ANALYST as a control variable. (A 

control variable -- as distinct from the control group — is one which 
holds some variable constant so, that the effects of other variables can 
be seen.) By controlling for analyst, one also controls for other 
variables correlated with analyst; in this case, variables such as the 
sex, experience ani style of the search analysts, the different ages of 



their users and so forth. It is apparent from the tables that the 
users of one analyst, Analyst B, reported learning more and being more 
satisfied than those of the other analyst. It is not our purpose to 
explain this fact and we would note that the differences may reflect as 
much differences in the users as in the analysts. 

Table 6 indicates that the asking of open and closed questions 
did not have a direct effect on the amount learned by users. As well, 
there was no significant interaction between analyst and question type; 
that is, asking open questions did not "work" better for one analyst while 
asking closed questions "worked" better for the other. 

Table 6 also indicates that the type of pause (moderate or no 
pause) did have a significant effect on the amount users learned, though 
not in the direction hypothesized. In particular, the interviews during 
which analysts tried to extend pauses to ten seconds produced users 
who learned significantly less than did interviews during which no pauses 
were used or when there was no attempt to change the natural style of 
interviewing. 

Table 7 reports the effects of the type of question and pause on 
overall satisfaction. .None of the differences, even between analysts, 
are statistically significant at the .05 level, the usual criterion used. 
We would note, though, that the p-values are relatively low for the main 
effects (i.e., for question and pause but not the interaction effects). 
Average satisfaction was higher when open questions were asked than in 
the control interviews or when closed questions were asked. As well, 
overall satisfaction was lower when moderate pauses were used than when 
no pauses werr allowed or when a natural interview style was used. 



One feature also evident in tables 6 and 7 are the small sample 
sizes in each cell, numbering as small as 5 and with a maximum of 14. All 
were supposed to be 15 (and if they had been we suspect all of the effects 
noted above would be statistically significant). ^Unfortunately, though 
all 150 searches planned were conducted, it was not possible to obtain 
evaluation results from all users. Further, some who did return 
evaluation questionnaires or who were interviewed on* the topic by phone, 
did not answer all items included in the satisfaction scale. 

Further analysis, reported later, helps to confirm, though, that 
the trends described above are, in fact, real. 

Importance of Information 

The key elements of the second research question concern relation- 
ships among the importance of information to a user (i.e., its value or 
worth), the amount of new knowledge that an individual gained as a result 
of conducting an online search, and his or her satisfaction with the 
information received as a result. Using a framework from the economics 
of information and decision theory, a rank ordering of satisfaction for 
different levels of importance and of information gain was hypothesized 
(figure 1 ) . 

Table 8 presents the average satisfaction scores for the four 
cells suggested by figure 1, and figure 2 presents the rank ordering of 
the cell means as actually observed. 



Amount Learned 
High Low 
High 2 4 

Importance 

Low 1 3 

Figure 2. Actual rank ordering of satisfaction 
scores in relation to the amount and 
importance of information obtained. 

The ordering is certainly not as hypothesized and, at first 
glance, appears to be unrelated to the original propositions. However, 
this is not the case* What was incorrect, apparently, was our expectation 
that importance would be the dominant independent variable. In ,fact, 
the amount learned dominates. Thus, those learning the most about their 
topic were, overall, more satisfied, than those who learned less. And, 
within these categories, those placing high importance on the information 
obtained tended to be associated with lower satisfaction scores rather 
than higher satisfaction scores, and vice versa. Stated simply, those to 
whom information is very important are harder to please. 

i 
\ 

Path Analysis 

A more sensitive test of the relationships treated in the previous 
two sections is possible using path analysis, a technique that allows the 
introduction of more independent and control variables without the loss of 
precision that would accompany comparable analysis using analysis of 
variance. 

Figure 3 presents the path diagram used *to assess the effect of all 
of the variables associated with the economics of information, framework 
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and the effect of the question experiment. Because the question and pause 
experiments were conducted separately, two separate path analyses must be 
conducted. In dojng so, one replicates many of the relationships apparent 
in figure 3. 

In the path .analyses, the control level of the experimental 
variables has to be dropped so that the question and pause variables be- 
come dichotomies. In the case of QUEST, closed questions were coded as 
0 and open questions as 1. Hence, positive path coefficients imply a 
relationship in which there is a positive effect of open questions on the 
dependent variables. For PAUSE, H no pause" is coded 0 and "moderate 
pause" as, 1 . 

In addition to considering the effects of all the independent and 
intervening variables on overall satisfaction (SATISFAC), analyses have 
also been conducted to determine their effects on individual items in the 
satisfaction scale. 

Table 9 presents the correlation matrix, means and standard 
deviates for all of the variables included in the first path analysis 
pictured in figure 3. The mean of 0.50 for ANALYST implies that 50% -of 
the searches were conducted by each analyst; the mean of 0.556 for QUEST 
implies '55 .6% of the interviews were part of the open question experi- 
mental group. Scale means for the other items are explained further in 
Appendix G. The sample size is 36, .indicating that 24 subjects were lost 
due to non-responses or the inability of researchers to contact the users. 
The number is smaller than that reported even for the two experimental 
groups earlier because a larger number of scales are included in this 
analysis. Each time a user skipped a key question, the sample size drops 
by pne. As well, a singleitem to measure knowledge gain was used in the 
earlier analysis whereas a two item scale is used here. 

3, o 
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Four zero-order correlations are statistically significant — those 
between ANALYST and PREKNOWL, between QUEST and "NEWKNOWL, between QUEST and ' 
SATISFAC, and between NEW KNOW L and SATISFAC. The first correlation (-0.27) 
indicates that the users of search analyst B (coded 1) reported that they 
were less knowledgeable about their topics at the start of their searches 
than did those of search analyst A (coded 0), This conclusion is consistent 
with the differences in the two analysts users, as noted earlier. 

The second pair of correlations (0.35 and 0.23) indicate that open 
questions were positively associated with both the user's gain in knowledge 
and overall satisfaction. The former of these is rather inconsistent with 
the finding reported earlier in table 6. The differences in the number and 
identity of the subjects included in the sample must account for this 
anomally. The other correlation is consistent with the trend noted earlier. 

The final correlation of 0.39 confirms the positive relationship 
between the amount a person learned and his or her overall satisfaction, a 

relationship identified in our earlier research. ) 

I 

These are, of course, zero-order correlations; that is, the effects 
of other variables have not been controlled. However, the path coefficients 
reported in figure 3 do reflect the strengths of relationships with the 
effects of all variables to the left of a given variable being controlled. 
Thus, the .39 coefficient for the path from QUEST to NEWKNOWL indicates 
that question type had a moderately strong effect on the amount a person 
learned, controlling for prior knowledge (PREKNOWL), ANALYST, and IMPORT 
(importance of information). The only' other significant effect was that 
of NEWKNOWL on SATISFAC. Though not significant, the coefficients of .24 
and .22 connecting ANALYST and QUEST to SATISFAC suggest residual effects 
Of these two variables on satisfaction independent of their effect through 



NEWKNOWL.-- Also note the unanalyzed correlations between ANALYST and PREKNOWL 
(noted earlier), between PREKNOWL and IMPORT, between JQUEST and IMPORT; and 
the slight negative effect of IMPORT on SATISFAC. While these are not 
statistically significant, they were found consistently throughout the 
analyses, suggesting they may be real. The correlation between QUEST and 

0 

IMPORT is by chance (since the question type was randomly assigned) but 

suggests that clients in the open question experimental group happened to have 

a greater need for information than did those in the^closed question group. 

Also, the -0.11 path coefficient between IMPORT and SATISFAC suggests that 

those with a greater need for information were harder to please, a relation-' 
ship also noted earlier. 

The f, e ,f or error variables in figure 3 represent hypothetical 

variables that would explain variation not explained in the current path 

model . 

In short, then, this analysis gives strong support for aspects of 
both the research questions. Open questions were more effective in 
facilitating searches that helped users learn more and gave them greater 
satisfaction, once relevant variables are controlled. Also, users for 
whom uncertainty was reduced (i.e., who learned more), were more satisfied. 
The importance or value of the information did not have a statistically 
significant eftect, but indications are that its effect may be the opposite 
of that anticipated. 

The path coefficients for the model in figure 3 are repeated in 
table 11. In addition coefficients are given for similar path models with 
the five items that compose the satisfaction index serving separately as 
dependent variables. Note that the first six coefficients are in fact 
zero-order correlation coefficients . In theory, they ought all be the same 
for all analyses, but in fact vary slightly because the samples change 
slightly from one analysis to the other. 



There are several consistent effects apparent in the item analyses 
that were not apparent in the path analyses of overall satisfaction. First, 
the>analyst'£ direct effect on satisfaction is apparent only in regards to 
the value of the bib! iography ■ (VALUEB) and its currency (CURRENT). Second, 
the direct effect of question type on satisfaction is apparent for the 
value of the bibliography, materials and utility, and not length or currency. 
Third, the effect of knowledge gain on satisfaction is apparent primarily 
for the value of materials (V ALU EM) and the utility of the search -resul ts . 
Fourth, the level of prior knowledge (PREKNOWL) has a negative effect on 
currency, and importance (IMPORT^ has a negative effect on length and 
utility. 

Taken together these relationships suggest that the links among question 
type, knowledge gain, and satisfaction with the bibliography, materials, 
and utility of results form the fundamental structure explaining satisfaction. 
The identity of the analyst, prior knowledge of the topic and the importance 
of the information tend to affect satisfaction with less critical areas such 
as the length and currency of the bibliography. 

The correlation matrix for the overall analysis involving the pause 
experiment is given in table 12. In general, the pattern of significant 
correlations is similar to that for the question experiment. However, the 
correlation between PAUSE and NEWKNOWL is not si gni ficant whil e that for 
QUEST and NEWKNOWL was, and the correlation between NEWKNOWL and PREKNOWL 
is significant where as it was not before. Correlations between satisfaction 
and PAUSE and NEWKNOWL show significant results whereas earlier analyses 
had suggested only trends. 
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The mean of 0.61 for ANALYST indicates the second search analyst 
conducted 61% of the search on which this analysis is based. The mean of 
13.39 for PAUSE means 39% were moderate pause interviews and 61% were no 
pause interviews. 

Table 12 reports the full set of path coefficients. Among the 
correlations, only the high negative correlations between ANALYST and 
PREKNOWL stand out, again emphasizing that the second analyst served 
primarily undergraduate and master of education students, while the first 
search analyst served many doctoral students. Path analysis does control 
for this difference in the two settings (as did controlling for analyst 
in the earlier analyses). 

Among the path coefficients, a consistent positive effect of prior 
knowledge on new knowledge is observed. PAUSE (i.e., moderate pauses) 
has negative effects on overall satisfaction and all satisfaction items 
except currency. PREKNOWL tends to have a negative effect on satisfaction 
(a relationship not suggested in the preceding analysis). IMPORT has a 
weak (not statistically significant) negative effect on satisfaction—similar 
to that . noted previously. Finally, NEWKNOWL, the gain in knowledge 
about the topic, has a strong positive effect on all aspects of satisfaction 
except currency. Again, note that analyst had a positive effect on currency 
(meaning the second analyst must have provided mere up-to-date searches 
in the eyes of the users). 
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Discuasion 

- In this study, we set out to explain a relationship we had identi- 
fied in earlier work — that between the amount a search service user 
learns, about his or her topic as a result of a search and the user's 
overall satisfaction. Two experiments were conducted, one in which the 
interviewer asked open or closed questions, and One in which the interviewer 
interjected pauses of moderate length or did not pause during the search 
interview. Finally, an explanation for user satisfaction was proposed 
that depended on inferences derived from the economics of information. The 
findings only partially verified the expectations set forth before the study 
was conducted. 

On the positive side, the experiment involving the use of open 
and closed questions provided strong evidence that interviews in which 
more open questions were asked resulted in searches in which users 
probably learned more and certainly weremore satisfied. Also, it proved 
that search analysts can be successfully trained to change their interview 
styles — even in a brief, one-day workshop — and that this training can 
result in a better bibliographic search service. 

The study also gives some support to a calculus of satisfaction 
in which prior knowledge, knowledge gain, and the importance of the infor- 
mation play a role. Certainly, the amount learned -- which we take as a 
measure of the reduction in uncertainty is strongly related to 
satisfaction. The importance of information — which we take as a measure 
of proximity of the searcher to the decision point — appeared not so 
important and, in fact, was negatively related to satisfaction. This 
finding suggests those near the point of decision are harder to satisfy 
than are those who have essentially already decided. This makes sense, 
though it was not what we had anticipated. 



Perhaps the greatest surprise was the strong negative effect that 
pauses had on the quality of searches. Recall that our intent was to 
slow down the interview process, to give the users time to express ideas. 
Too often, it seemed to us, search analysts dominated, directed and 
decided, sometimes cutting off users in order to assert their own 
understanding of the users 1 topics. 

In translation to action, something may have been lost. Perhaps 
mandating 10 second pauses versus no pauses was too extreme. As well, 
there was the problem, for both the interviewers and us, of knowing when 
a pause occurs. Is it after a search analyst speaks? After the client 
speaks but before the analyst speaks? A break in the talk of either 
participant? We found operational izing the concept difficult. Our decision, 
described earlier, seemed the best choice, but may have missed something 
vital. Certainly, as the validation of the treatment and assumptions make 
clear, the analysts did pause longer when they were supposed to; sometimes, 
it appears, they paused too long. 

It would seem , then, that pauses do make a difference. Longer 
pauses, especially if the analyst is not paying attention to the user, 
have negative effects. This is not what we had intended to demonstrate, 
but nevertheless is a finding that has been clearly demonstrated. As 
well, it is apparent that it is harder to train analysts to use pauses 
effectively than it is to train them to use open and closed questions 
effectively. 

Finally, the dominance of the "amount learned 11 , and the weakness 
of the "importance of information" in determining satisfaction suggests, 
for the users studied, that the decision-making model concentrating on 
the reduction in uncertainty \s not fully appropriate. Indeed, it may 
have been inappropriate. 

4. 
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It was suggested earlier that a reversal of the hypothesized roles 
of knowledge gain and the importance of information in explaining satis- 
faction would corroborate of a model of satisfaction based on cognitive 
psychology rather than the decision-making model proposed. Perhaps, then, 
the importance of the amount learned implies that the contribution of the 
search and its products to the ideas, understanding, and thinking of the 
user is of utmost importance, and not its immediate utility for decision- 
making in an economic context. This interpretation would be consistent 
with the fact that most searches conducted by the users of the analysts 
in this study were for the purposes of assignments. That is, the users 
were students who were learning, not consumers or administrators making 
decisions. 



It has been said that more is known about white rats and first 
year psychology students than any other two creatures on earth. Perhaps 
university library clientele should be added to this list. 

The major limitation of the current study is the restricted 
nature of the cl ientel e served by the two search analysts. Virtually all 
users were university students in education faculties who walked in to 
their respective libraries and requested searches on single topics, usually 
related to an assignment that they were doing. 

We suspect that the basic effects of the experiments and of the 
amount learned on the users' levels of satisfaction would be observed 
for populations other than education students. However, the relative roles 
of "prior knowledge" and the "importance of the information" might be very 
different for other populations seeking information for other reasons. 





Limitations 




eric 
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That there were only two search analysts and, in the end, very 
small . sample sizes might also be considered limitations. Yet, the major 
effects were strong enough to be seen in spite of the different styles or 
approaches of the search analysts and the small sample sizes.. If the 
effects are strong enough to be evident in such a hostile field setting^ 
we believe that they would.be evident in a larger scale study that would 
overcome* these limitations. 

Finally, we note that there were "constants" in many places where 
there could have been variables, and that many nuisance variables were not 
controlled but assumed to be randomly distributed. For example, method of 
contact (walk-in), type of search (retrospective) and type of library 
(education) were all constants. System used, topic, data base, approach 
to searching (e.g., whether or not the user was present during the search) 
were all assumed to vary randomly, though they could have been introduced as 
independent variables, given a sufficiently large sample size. 

Implications and Conclusions 

The findings of this study are quite promising as far as the practice 
of online bibliographic searching is concerned. The search interview has been 
confirmed as a maleable factor that affects the quality of the search, as 
perceived by the user. Not only can one conclude that search analysts 
would do well to develop a facility to ask open questions, but that other 
factors that have been identified in the literature on interview techniques 
may prove useful as well. 

The negative effects of moderate pauses in one of the experiments 
indicates that caution is necessary, however. Identification of factors 



that can be manipulated to affect satisfaction carries with it the 
implication that negative as well as positive effects may occur. 
Cautious experimentation and continuing evaluation is necessary to find 
the right combi nation . 

As far as the pause experiment is concerned, one should not 
conclude that any attempt to slow the pace of questioning will reduce 
satisfaction, as was the case here. One would, however, be advised to 
use shorter pauses. Were we to repeat this study, we might be more likely 
to advise the search analysts simply to "count to 3 H after a break in 
conversation to ensure the user has nothing more to say. 

The evidence concerning the utility of the framework drawn from 
the economics of information to explain user satisfaction is more 
ambiguous. The reassessment of the relative influence of two key variables 
— the importance of information and the amount learned — in light of 
the findings may be appropriate. That is, if the reaction in uncertainty, 
as measured by the amount learned, has a more powerful inflt^nce on the 
satisfaction of users than does the nearness of the user to- his or her 
decision-point, as measured by the importance of the information, then the 
findings make sense. Even then, many of the relationships are weak and 
somewhat unexptected; e.g., the fact that those with greater prior knowledge 
or for whom information is more important express lower satisfaction, 
implying that they are harder rather than easier to satisfy than others, 
even when the information they obtain is relevant to their needs. 

Alternatively, these findings may indicate the proposed framework 
is inadequate, and that a better explanation, at least for the user 
of the services studied here, is to be found elsewhere. One approach that 
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tias been suggested draws on cognitive psychology and emphasizes the 
"maps" that people develop in their minds in order to account for the 
world they observe (Woodward, 1980). Such an alternative explanation 
would be consistent with the findings, in which the amount of knowledge 
gained is a powerful intervening variable. It would also suggest, 
perhaps, new approaches to the strategies employed by interviewers in 
attempting to draw out the maps in their users 1 minds, and translate 
them into the artificial world of ANDs, ORs, and AND NOTs used by search 
analysts. 
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Figure 3. Path analysis model for explaining user 
satisfaction. 
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Table 1 Variables Used to Explain User Satisfaction 



NO . 


Variable 
Name 


Variable Description 
and Coding 


1 . 


ANALYST 


Search analyst A - 0 
Search analyst B - 1 


2A. 


QUEST 


Closed Question - 0 
Open Question - 1 
Control (not used - 3 
in path analysis) 


2B. 


PAUSE 


No pause - 0 
Moderate pause - 1 
Control (not used - 3 
in path analysis) 


3. 


PREKNOWL 9 


Extent of knowledge about 
topic before search 


4. 


IMPORT 9 


Importance of information 
being sought 


5. 


NEW KNOW L a 


Increase in knowledge about 
topic due to search 


6. 


SATIS FAC a 


Satisfaction with search 



See Appendix G for complete list of scale items used, 
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Table 2 Characteristics of Users of Search Services by Analyst 



Analyst 

Characteristic 

A B Total /Overall 



; F > - — — 

User ' s organization 








School system 


1 .4% 


1 .4% 


1 .4% 


Community college 


2.6 


1 .4 


2.0 


Faculty of ed. 


0.0 


5.7 


2.7 


Ministry of ed. 


0.0 


1 .4 


0.7 


University 


96.1 


90.0 


93.0 


User's sex 








Male 


45.5% 


48.6% 


47.0% 


Female 


54.5 


51 .4 


53.0 


User's age 








25 or under 


12.9% 


46.6% 


29.2% 


26 - 35 


43.5 


36.2 


40.0 


36 - 45 


30.6 


10.3 


20.8 


over 45 


12.9 


6.8 


9.9 


User's highest degree 








None 


3.2% 


29,3% 


15.8% 


Bachelors 


22.6 


43.1 


32.5 


Masters 


56.5 


17.2 


37.5 


Doctorate 


16.1 


10.3 


13.3 


Othpr 


1 .6 


0.0 


0.8 


User's role 








Administration 


3.2% 


3.4% 


3.3% 


Teaching 


11 .1 


8.6 


9.9 


Research 


9.5 


10.3 


9.9 


Undergraduate 


1 .6 


25.9 


13.2 


Masters 


25.4 


32.6 


29.0 


' Doctoral 


46.0 


8.6 


28.1 


Other 


3.2 


10.3 


6.8 
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Table 3 Characteristics of Searchers Conducted by Analyst 



Analyst 



Characteristic 


A 

(n ■ 77 )« 


B 

(n - 75) 


Total /Overall 


Purpose 








Keep abreast 


1 AX 


1 AX 


1 .4* 


Assignment 


OO A 

82 .4 




7fi ? 

/ 0 .C 


. Bibliography 


0.0 


1 7 

2.7 


1 .4 


Curric. Dev. 


n 7 
2.7 


ft ft 




Program Imp. 


u . u 


9 7 
C.I 




Speech, article 


0.0 


5.5 


9 7 

2 . / 


R&D 


13.5 


12.3 


TO ft 

12.9 


Other 


n ft 

0.0 


c c 

D .9 


9 7 
2 . / 


Contact method 








Walk-In 


1 ftft ft w 

100.01 


no CW 

9a. o* 


yy .j* 


Letter 


0.0 


1 A 

1 .4 


n 7 
u . / 


User supplied 








Synonyms 


86.81 


83. 3% 


85 .1 % 


Logic 


2.6 


16.7 


ft c 

9.5 


Other references 


2.6 


' 0.0 


1 .4 


Interview time 


15.7 min. 


5.4 min. 


12.3 min. 


Strategy time 


11.2 mln. 


ID 7 m-tfl 

I a . / min. 


17 7 m-t n 

1 J . / nil n . 


Connect time 


14.3 mln. 


17.4 mln. 


1 b .o mi n . 


Total search time 


41 .2 mln. 


41 .5 min . 


41 .o mi n . 


System used 








SDC 


o.ox 


O.OX 


u .ux 


LIS 


22.4 


23.2 


99 Q 

22.0 


QL 


0.0 


ft ft 
0.Q 


u.u 


INFOGLOBE 


0.0 


2.9 


1 A 

I .4 


BRS 


97 .4 


QQ A 


30 ■ 1 


OTHER 


1 .4 


2.9 


2 . 1 


Data base 








ERIC 


90. 9% 


36 ,5X 


04 .cm 


PSYCHA8S . 


20.8 


35.1 


97 0 

2/ .0 


SOCABS 


18.2 


C A 

5 .4 


11 0 

1 1 .y 


SSCI 


ft ft 
u.u 


10 .£ 


7.9 


EXcep. Child 


5.2 


9.5 


7.3 


DISSABS 


9.1 


14.9 


11 Q 


OTHER 


36.4 


75.1 


55.6 


• Citations printed 








Online 


0.6 


30.0 


14.0 


Offline 


86.8 


62.6 


75.0 


Total 


87.4 


92.6 


89.0 


Price 


t 3.34 


$16.08 


$ 9.71 


<User present for search 


9.2% 


98. 6X 


52. 7X 


Interaction during 
search 


7.9X 


82.8X 


42. H 



1 Different numbtrs of clients responding to various Items. 
Two extra starches not included In other analysts. 
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Table 4 Verification of Experimental Treatment 



Question Proportion of Open Questions 

Type Mean S.D. 

Control 16.5 10.7 

Open 23.9 17.6 

Closed 12.9 7.9 



F(2,87)=5.819 P*.004 



Pause Average Length of Pause 
Mean S .D. 

Control N/A N/A 

Mod. Pause 2.06 1.50 

No Pause 0.68 0.74 

F(1.,58)=20.620 P^.001 

* \ 

) 



Table 5 Validation of Effect of Questions and Pauses on Responses 



Response 

Characteristics Question Pause 

Open Closed No Pause Pause 

( n =48) (n*50) (n=10) (n=10) 



No. 


of words x 


30.6 


12.5 


17.8 


8.6 




s 


25.1 


18.1 


37.3 


8.3 


No. 


of seconds x 


15.6 


6.4 


8.5 


4.0 




s 


12.4 


11.4 


17.6 


5.0 


No. 


of concepts x 


2.4 


1 .3 


1 .9 


1 .4 




s 


1 .4 


0.8 


1 .8. 


0.7 




Table 6 Summary of ANOVA Results for Assessing Affects of 
Question Type and Pause on New Knowledge 
Controlling for Analyst 



Question Experiment 
Cell Means and Frequencies 





Pnn+rnl 


Dnpn 






Analyst A 


7.00 


6.57 




7 .09 




(13) 


(14) 




(11) 


Analyst B 


7.64 


7.91 




7.31 




(14) 


(ID 




(13) 


Total 


7.33 


7.16 




7.21 




(27) 


• (25) 




(24) 


QUEST 


F(2,70) = 0 


.05 


P 


= .948 ; 


ANALYST 


F(l,70) = 3 


.66 


P 


= .060 


Q x A 


F(2,70) = 1 


.95 


P 


= .501 



Pause Experiment 
Cell Means and Frequencies 





Control 


Moderate 


None 


Analyst A 


7.00 


5.33 


7.15 




(13) 


(9) 


(13) 


Analyst B 


7.64 


7.11 


8.00- 




(14) 


(9) 


(11) 




7.33 


6.22 


7.54 


Total 


(27) 


(18) 


(24) 


PAUSE 


F(2,53) = 


3.80 p = 


.014 


ANALYST 


F(l,63) = 


3.14 p = 


.050 


P x A 


• F(2,63) = 


1 .86 p = 


.572 
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Tafale 7 Summary of ANOVA Results for Assessing Affects of 
Question Type and Pause on Overall Satisfaction 
Controlling for Analyst 



Question Experiment 

Cell Means and Frequencies 

Control Open Closed 

Analyst A 17.8 18.8 16.6 

(12) (13) (10) 

Analyst B 18.9 20.6 18.0 

(14) (10) (12) 

Total 18.4 19.6 17.4 

(26) (23) (22) 



QUEST F(2,65) = 2.008 p = 0.143 

ANALYST F(l ,65) = 2.301 p = 0.134 
Q x A F(2,65) = 0.818 p = 0.949 



Pause Experiment 
Cell Means and Frequencies 





Control 


Moderate 


None 


Analyst A 


17.8 


16.6 


17.6 


(12) 


(5) 


(11) 


Analyst B 


18.9 


16.9 


19.6 


(14) 


(8) 


(11) 


Total 


18.4 


16.8 


18.6 




(26) 


(13) 


(22) 


PAUSE 


F(2,55) = 


1,320 p = 


0.276 


ANALYST 


F(l,55) = 


1.728 p = 


0.194 


P x A 


F(2,55) = 


0.224 p = 


0.800 



ERIC 
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Table 8 Analysis of Variance Showing Relationship of 
Importance of Information and New Knowledge 
with Overall Satisfaction 



Cell Means and Frequencies 

NEWKNOWL 







High 


Low 


Mean 
(Total ) 




High 


19.1 
(29) 


15.7 
(23) 


18.3 
(53) 


a 

IMPORT 












Low 


19.8 
(27) 


16.8 
(26) 


17.6 
(52) 




Mean 


19.4 
(56) 


16.3 
f49) 


18.0 
(105) 




ANOVA Table 






Source of Variation 


SS 


af 


MS 


F p-value 


IMPORT 


21 .03 


1 


21 .03 


1 .460 0.230 


NEWKNOWL 


261 .32 


1 


261 .32 


18.132 0.000 


IMPORT x NEWKNOWL 


1 .48 


1 


1 .48 


0.103 0.749 


Expl ained 


277.30 . 


3 


92.44 




Resicual 


1455.61 


101 


14.41 




Total 


1732.91 


104 


16.66 





a Item (a) in the IMPORT scale (Appendix G) was used to measure IMPORT. 
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Table 9 Correlation Matrix, Means, and Standard Deviations 
For Variables in Question Experiment 







X l 


X 2A 


X 3 


X 4 


X 5 X 6 


ANALYST 


1 












fit 1 r-r» t 

QUEST 


X 2A 


-0.112 










PREKNOWL 


X 3 


h 

-0.274° 


0.235 








IMPORT 


X 4 


0.131 


0.225 


-0.272 






NEWKNOWL 


X 5 


0.095 


0.354 b 


-0.048 


0.217 




SATIS FAC 


X 6 


0*232 


0.281° 


-0.125 


0.002 


0.393 3 


Mean 


0.500 


0.556 


8.81 


13.97 


6.44 18.00 


Std. 


Dev. 


0.507 


0.504 


1.95 


3.66 


1 .78 3.63 


n = 


36 













3 0.01 
b p&0.05 



Table 10 Correlation Matrix, Means, and Standard Deviations 
for Variables in Pause Experiment 







X l 


X 2B 


X 3 


X 4 


X 5 X 6 


ANALYST 


X l 












PAUSE 


X 2B 


0.088 










PREKNOWL 


X 3 


-0.523 3 


-0.125 








IMPORT 


X 4 


-0.056 


-0.015 


0.162 






NEWKNOWL 


X 5 


-0.172 


-0.128 


0.304° 


0.247 




SATIS FAC 


X 6 


0.117 


-0.301° 


-0.138 


-0.173 


0.405 a 


Mean 




0.613 


0.387 


8.81 


14.42 


6.90 18.00 


Std. 


Dev. 


0.495 


0.495 


1.97 


4.75 


2.02 4.02 


\n = 


31 













E^C ' p£0 ' 01 



5c» 
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Table 11 Path Coefficients for Assessing Affect of 
Question Type on User Satisfaction 



Satisfaction Item 





Path 


Statistic 


SATISFAC 


LENGTH 


VALUEB 


VALUEM 


CURRENT 


UTILITY 




ANALYST-QUEST 


12 


-0.11 


-0.08 


n no 


n no 

_ -0*08 


-CL1 1 _ 




2. 


ANALYST- PREKNOWL 


1 3 


-0.27 


-U. oc 


-U.o£ 


n 99k 

-U . OC 


n 97 

- U . C / 


H 90, 


3. 


ANALYST- IMPORT 


14 


0.13 


C\ 1 9 

U . 1 o 


n i 9 


fi 1 9 


u . 1 0 


w.l/ 


A 


QUE5T-PREKN0WL 


23 


-0 .£4 


-U .£0 


n 9*3 
-U . co 


n 9*3 

-U .£0 


-U • CH 


- U . £ / 


5 . 


QUEST-IMPORT 


cQ 


U . to 


n 90^ 

U • CD 


n 9Q^ 
u , c y 


n 

U • C 3 


U • CO 


0 26 b 


6. 


PREKNOWL- IMPORT 


34 


-0.2/ 


-U .£D 


n 9K 


n 9£ 

-U .CO 


- U • c I 


n pq^ 

-U . 


7. 


NEWKNOWL-ANALYST 




0.16 


U . 1 C 


n i 9 

U . 1 C 


U . 1 C 


U . 1 0 


n n 
u . 1 0 


8. 


NEWKNOWL-QUEST 


P52 


0.39 a 


0.38 a 


0.38 a 


0.38 a 


0.39 a 


0.35 a 


• 


NEWKNOWL- PREKNOWL 


P53 


0.08 


0.09 


0.09 


0.09 


0.09 


0.06 


10.' 


SATISFAC- ANALYST 


Pei 


0.24 


0.02 


0.18 


0.17 


0.26 


0.11 


11. 


SATISFAC-QUEST 


P62 


0.22 


0.01 


0.25 


0.21 


0.00 


0.28 b 


12. 


SATISFAC- PREKNOWL 


P63 


-0.04 


0.01 


0.14 


0.03 


• -0.22 


0.01 


13. 


SATISFAC- IMPORT 


P64 


-0.11 


-0.17 


0.00 


-0.04 


-0.02 


-0.14 


14. 


SATIS FAC-NEWKNOWL 


P65 


.33 b 


0.26 


0.17 


0.41 a 


0.20 


0.41 a 






n 


36 


39 


39 


39 


36 


38 






x 


18.0 


3.59 


3.59 


3.36 


3.89 


3.58 






s 


3.63 


0.88 


0.94 


0.93 


0.85 


1 .03 




a 
b 



p4 0.01 
p^. 0.05 
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Table 12 Path Coefficients for Assessing Affect of 
Pause Length on User Satisfaction 



Variable 





patn 


OlaL I b 1 1 L 




1 ENGTH 


VALUEB 


VALUEM 


CURRENT 


UTILITY 


1 . 


ANALYST-PAUSE 


r 12 


0.09 


0.04 


0.04 


0.04 


0.09 


0.04 


2. 


ANALYST- PREKNOWL 


r 13 


-0.52 3 


-0.49 a 


-0.49 9 


-0.49 a 


-0.52 a 


-0.49 


3. 


ANALYST-IMPORT 


r 14 


-0.06 


-0.03 


-0.03 


-0.03 


-0.06 


-0.03 


4. 


PAUSE-PREKNOWL 


r 23 


-0.13 


-0.14 


-0.14 


-0.14 


-0.13 


-0.14 


5. 


PAUSE-IMPORT 


r 24 


-0.01 . 


-0.04 


-0.04 


-0.04 


-0.01 


-0.04 


6. 


PREKNOWL-IMPORT 


r 34 


0.16 


0.17 


0.17 


0.17 


0.16 


0.17 


7. 


NEWKNOWL- ANALYST i 


P51 


-0.01 


-0.03 


-0.03 


-0.03 


-0.02 


-0.03 


8. 


NEWKNOWL-PAUSE 


P 5 2 


-0.09 


-0.08 


-0.08 


-0.08 


-0.09 


-0.08 




N tWKNUWL -rKtNlNUWL 


p 53 


0 29 


0.28 


0.28 


0.28 


0.29 


0.28 


# 


SATISFAC-ANALYST 


Pel 


0.09 


0.16 


-0.10 


-0.16 


0.39 b 


0.10 


11. 


SATISFAC-PAUSE 


P 62 


-0.28 b 


-0.30 b 


-0.24 


.-0.41 9 


0.06 


-0.21 


12. 


SATIS FAC-PREKNOWL 


P 63 


-0.24 


-0.26 


-0.22 


-0.31 b 


-0.12 


-0.11 


13. 


SATISFAC-IMPORT 


P64 


-0.15 


-0.26 


-0.18 


-0.18 


0.22 


-0.26 


14. 


SATIS FAC- NEWKNOWL 


P65 


0.52 9 


0.42 a 


0.54 a 


0.36 a 


0.21 


0.57 a 






n 


31 


32 


32 


32 


31 


32 






X 


18.0 


3.50 


3.50 


3.59 


3.74 


3.69 






s 


4.02 


0.76 


1 .11 


1 .07 


0.97 


1 .03 



9 p£ 0.01 
b p< 0.05 
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Appendix A 



The Ontario Institute for Studies in Education 
Department of Educational Administration 

SSHRC Project: Search Interview Techniques for 
Online Bibliographic Retrieval 

TRAINING SESSION FOR SEARCH ANALYSTS 
Held Nov. 14, 1979, 10am-4:30pm 
01 SE, Room N733 



SCHEDULE OF ACTIVITIES 

10:00 am Introduction of Participants 

10:15 am Overview of the Project 

Purpose 

Types of searches 
User sample 

Instruments for Data Collection 

- Search Analyst's Record 

- Taped Interviews 

- User Questionnaire 

Sequence of Experimental Interview Techniques 

11:15 am The Search Analyst's Role in the Research 

Experimental Interview Techniques 

- open questions (o) 

- closed questions (c) 

- ten - second pauses (p) 

- no pauses (n) 

- control 



12:00 pm LUNCH 

1 : 30 pm Simulating Experimental Interview Techniques 

3:00 pm Analysis oT Simulated Interviews 

4:30 pm Wrap-Up 



Materials for Distribution: CLJ, JASIS articles 

Sequence for experimenta i i nterviews 
Search record 
User questionnaire 
Tape cassettes 



ERjfc b'j 
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Appendix B 

Sequence of Experimental Interview Techniques 
Search Analyst A 



No. 


Technique 


No. 


1-15 


control 


35 


16 


0 


36 


17 


n 


37 


18 


n 


38 


19 


P 


39 


20 


0 


40 


21 


n 


41 


22 


0 


42 


23 


n 


43 


24 


n 


44 


25 


P 


45 


26 


0 


46 


27 


P 


47 


28 


0 


48 


29 


0 


49 


30 


0 


50 


31 


p 


51 


32 


c 


52 


33 


n 


53 


34 


c 


54 



Technique No. Technique 

p 55 o 

c 56 c 

c 57 p 

o 58 o 

c 59 n 

c 60 p 

p 61 p 

c 62 c 

p 63 n 

n 64 o 

n 65 n 

o 66 c 

p 67 n 

o 68 p 

c 69 "c 

n 70 o 

n 71 c . 

c 72 c 

o 73 n 

p 74 p 

75 p 



9 
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Appendix C 

Sequence of Experimental Interview Techniques 
Search Analyst B 



Nn 


Tprh ni chip 


No. 


Technique- 


No. 


Techni 


1-1 5 


rnntrnl 


35 


c 


55 


0 


1 6 


n 
r 


36 


0 


56 


c 


1 7 


r 


37 


n 


57 


0 


1 o 


1 1 


38 


c 


58 


c 


1 Q 




39 


c 


59 


n 


?n 
cv 


n 
P 


40 


c 


60 


p 


91 


u 


41 


n 


61 


p 


99 


n 
u 


42 


c 


62 


p 


CO 


0 


43 




63 


n 


24 


n 


44 


0 


04 


n 


25 


. P 


45 


n 


65 


0 


26 


n 


46 


P 


66 


0 


27 


P 


47 


n 


67 


0 


28 


0 


48 


c 


68 


n 


29 


0 


49 


c 


69 


0 


30 


n 


50 


P 


70 


P 


31 


P 


51 


c 


71 


0 


32 


c 


52 


P 


72 


c 


33 


n 


53 


n 


73 


p 


34 


n 


54 


c 


74 


c 










75 


p 
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Appendix D 



Search No. : m 

Ana lyst : 

Treatment: 



B 



1 . Name 



The Ontario Institute for Studies In Education 
Department of Educational Administration 

Search Analyst T s Record of Online Search 
3, Topic: ' 



2. Address 



Tel. ( 



) 



5. Purpose of search: 

Keep abreast of field 

Assignment, term paper, thesis 

Prep., update of bibliography 

^Curriculum Development 

Program improvement 

Speech, article, report 

6. Contact: Walk- in Letter 



4. Date (day/mo. /yr. ) , , t 



t t t t ? t 



User suppl ied; 



jsynonyms 



Interview time , , , t m ln. 
10. Connect time , , » f m in. 



Research/development report 

B rows i ng 

Personnel recruitment, evaluation 

Policy development 

Other: 



Phone Other: 

J°9' c o ther refs. 

9. Strategy time j t , t m ?n. 

1 1 . System: SDC _ 

Lockheed _ 

QL 

InfoGlobe 



none 



J3RS 

_N.Y. Times 
Other: 



12. Data base(s): ERIC _ 

Psych. Abs. 

Soc. Abs. 

13. Citations printed online , , , t , 



^SSCI 
_ECER 

Child Abuse 



JDiss. Abs. 
Other: 



14. Citations printed offline t f t t t 



15. Price charged: 



16. User present during search; 

Other: 



Yes 



No 



17. Did user interaction take place during search: 



18. Requestor T s organization; 



School board 



9. Sex: 



Male 



_CAAT 

Faculty of education 
Female 



Yes No 

Ministry of Education 

University 

Other: 



6: 



SA 



User 



Search No. : 

■ — 

Appendix E .56. Respondent : _ 

The Ontario Institute for Studies in Education 
Department of Educational Administration 

User Responses to Online Search Request 

The Social Sciences and Humanities Research Counci I of Canada has sponsored 
this inquiry into factors affecting user satisfaction v/ith online retrieval services. 
The data collected from this user questionnaire will be used only for research purposes, 
Ultimately, it is hoped that the results will help to improve online services. Please 
read each question carefully and indicate your response with a check mark: \/_ 



Personal P at a 

K. Age 



25 or under 
26-35 



36-45 



46-55 



_56-65 
over 65 



2, Highest earned degree 

None 

Bachelors 

Master's 

Doctorate 

Other : 



3. Please indicate your primary professions! role or function. (Check one only) 
Administration or supervision Undergraduate student 



_Teach i ng 



B.Ed, or M.Ed, student 



Research 



Other master f s student 



Library services 



Doctora I student 



Private consultant 



Other 



4. How many years have you been in your current position? 
Less than one year 4 to 6 years 

1 year 7 to 9 years 

2 years 10 or more years 

\5 years 

Extent of Knowledge About Topic Before Search 

5. Have you written any professional papers for publication on the topic of your 
search in the last five years? 



Yes 



No 



Have you given any talks or presentations at conferences on this topic in the 
last five years? 



Yes 



No 



7. Have you taken any graduate-level courses on this topic in the last five years? 
Yes No 

8. Have you tauoht courses on this topic in the last five years? 

Yes No 
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Extent of Knowledge About Topic Before Search (Cont'd) 

9. Have you participated In an research project on this topic in the 
last five years? 

Yes ' No 

10. How would you rate the amount of knowledge you had on this topic before the 
search? 

No knowledge* Advanced 



Basic Expert 

Moderate 

11. Please Indicate which of the following sources you had already consulted or 
searched before conducting the online search. 

Human resources Prof, development days 



_0ffice files, reports Academic, prof, courses 

_Prof. organizations Curriculum guides, texts 

Libraries and their holdings — ERIC materials 

journals, books, indexes 

* Other 

Conferences 



Extent of New Knowledge About Topic After Search 

12. How much did you learn about your topic as a result of the search? 

Nothing 

Very little 

Some 



Q uite a bit 
Extremely large amount 

Perc e ntage 

13. Of the citations retrieved that were Of 1-102 l0-20?I 21-50T More than 
relevant to your topic, how many were 50$ 
familiar to you prior to this search? a 

14. Of the citations retrieved that were fc - 3 
relevant to your topic, how many 

were new to you? ^ m 



15. How many of the citations 

retrieved do you plan to read? . 

16. How much additional information do you still desire concerning the topic that 
was searched? 

Nothing 

Very little 



ERIC 



Some 

puite a bit 

9p Extremely larpe amount £J. : 



Value Placed on New Knowledge gg 
Please indicate your satisfaction witfK Satisf action 



the following aspects of your search. Very Low Low Moderate High Very High 

|7. Helpfulness of search analyst 

18. Length of bibliography 

19. Time taken to deliver bibliography ^ 

20. Value of the bibliography 



21. Value of materials located via 
bibliography 

22. Currency (up-to-dateness) of 
the search results 

23. Utility of the search results 
for your intended purpose 

24. Price of the search 



Consequences of Missed Information 

Affect 

25. If your search had missed 10£ of rn — r Cl . . r ' . . — ^ -rr — 

1 . . ui Not a+ Slightly Some Consider- Great y 

a I I what ably 



the relevant citations available 
in the system, how would this 
affect the ultimate purpose for 
which you placed the search? 



26. If your search had missed 50% 

of the relevant citations available 
in the system, how would this 
affect the ultimate purpose for 
which you placed the search? 

27. If your search had missed 10£ of 
the relevant citations available 
in the system, how would you be 
affected financially? 

28. If your search had missed 50% of 
the relevant citations available 
in the system, how would you be 
affected financially? 



29. If your search had missed 10/S of the relevant citations, would the project or 
purpose for which you placed the search be undertaken or completed anyway? 

Yes No 



30. If your search had missed 50$ .of the relevant citations, Would the project or 
purpose for which you placed the search be undertaken or completed anyway? 

■ Yes No 
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31. If 50$ of the documents retrieved were irrelevant, v/hq v/ould be most affected? 

Y ou only \ 

t You and your work- group 

You and your entire organization 

_A larger number of*- people 

Not appl i cable 

32. If 90% of the documents retrieved were Irrelevant, who would be most affected? 
You only . 

Y our and your work group 

Y ou and your entire organization 

♦ 

A larger number of people 

Not appl i cable 

33. Comments: 



THANK YOU FOR YOUR COOPERATION 



Please return your completed questionnaire to: 

Dr. E.W. Auster, Principal Investigator, 

Department of Educational Administration 

The Ontario Institute for Studies in Education 

252 Rloor Street West, N752 

Toronto, Ontario 

M5S 1V6 

(416) 923-6641, Ext. 420 



9 
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SEARCH NO. 
ANALYST 



Appendix F 

TOTAL NO* OPEN QUESTIONS: ( 

TOTAL NO. CLOSED QUESTIONS: ( 

MEAN PAUSE TIME; 



analyst time user 



analyst time user 



ERIC / 



O = open 



C= closed S= statement R a response P= prompt 



oue&tion Question 



\ 
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Appendix G: Items Used in Scales 

PREKNOWL - Extent of Knowledge About Topic Before Search 

\ (a) Have you written any professional papers for publication 
on the topic of your search in the last five years? 
Yes - 2; No- 1 . 

(b) Have you given any talks or presentations at conferences 
on this topic in the last five years? 

Yes - 2; No - 1. 

(c) Have you taken any graduate-level courses on this topic 
in the last five years? 

Yes - 2; No - 1 . 

(d) Have you taught courses on this topic in the last five 
years? 

Yes - ii No - 1 . 

(e) Have you participated in a research project on this topic 

in the last five years? 

\ ■ 

Yes - 2;\No - 1 . 

(f) How would yot^ rate the amount of knowledge you had on this 
topic before the search? 



No knowledg^ - 1; Basic - : 2; Moderate - 3; 
Advanced - 4;^ Expert - 5. 



\ 



ERIC : \ i 
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IMPORT - Consequences of Missed Information 

(a) If your search had missed 10% of the relevant citations, 
how would this affect the ultimate purpose for which you 
placed the search? 

Not at all - 1; Slightly - 2; Somewhat - 3; 
Considerably - 4; Greatly - 5. 

(b) If your search had missed 50% of the relevant citations 
available in the system, how would this affect the 
ultimate purpose for which you placed the search? 

Same scale as (a) . 

(c) If your search had missed 10% of the relevant citations 
available in the system, how would you be affected 
financially? 

Same scale as (a) . 

(d) If your search had missed 50% of the relevant citations 
in the system, how would this affect you financially? 

Same scale as (a) . 

(e) If your search had missed 10% of the relevant citations, 
would the project or purpose* for which you placed the 
search be undertaken or completed anyway? 

Yes - 1 ; No - 2 . 

(f) If your search had missed 50% of the relevant citations, 
would the project or purpose for which the search was 
undertaken be completed anyway? 

Yes - 1 ; No - 2 . 



ERIC 7 X 
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(g) If 50% of the documents retrieved were irrelevant, who 
would be most affected? 

You only -1; You and your work group - 2; 
You and your entire organization - 3; 
Larger number of people - 4; 
Not applicable - recoded 1 for analysis. 

(h) If 90% of the documents retrieved were irrelevant, who 
would be most affected? 

(same scale as (g) 
NEW KNOW L - Extent of New Knowledge About Topic After Search 

(a) How much did you learn about your topic as a result 
of the search? 

Nothing - T; Very little - 2; Some - 3; 
Quite a bit - 4; Extremely large amount - 5. 
(c) How many of the citations retrieved do you plan to read? 
0% - 1; 1-10% - 2; 10-20% - 3; 21-50% - 4; 
more than 50% - 5. 
SATISFAC - Satisfaction with the following aspects of the search 
(Satisfaction very low - 1; Low - 2; Moderate - 3; 
High - 4; Very high - 5.) 
LENGTH: Length of bibliography 
VALUEB: Value of the bibliography 

VALUEM: Value of materials located via the bibliography 
CURRENT: Currency (up-to-dateness) of search results 
UTILITY: Utility of the search results for your intended purpose 
OVERALL: Sum of preceding five items. 

Note: Omitted from analysis were satisfaction with 
helpfulness of search analyst, time taken to 
deliver bibliography, and price of search. 



-64- 



PART II 

THE ONLINE NEGOTIATION INTERVIEW IN TRADITIONAL PERSPECTIVE: 
A REVIEW OF THE LITERATURE 

Introduction » 

The focus of this study is on the relationship between selected 
aspects of the negotiation process in online bibliographic retrieval 
and the ultimate satisfaction of the user with the search outcomes. 
The negotiation process itself, however, is not a single act but 
rather a series of complex interactions that may involve the search 
analyst and the client; the search analyst and the computer system; 
the search analyst and the user interacting with each other and the 
computer system. Further, the negotiation process may not only involve 
various human/machine combinations but is itself a transaction that is 
comprised of a series of component parts, not all of which need occur 
to conduct each search and the discrete parts of which still remain 
ill-defined. In addition, it is not yet clear how, or indeed whether, 
the negotiation process in online retrieval differs from the more 
traditional and more fully documented reference interview in the manual 
search for information. Nor is it clear what elements affect the user's 
ultimate satisfaction with the search, or how these factors may be 
separated out, measured, and related to the preceding elements of the 
search process. Therefore it would seem reasonable in this section of 
the report to begin by presenting the literature that attempts to describe the 
online negotiation process in general terms, then summarize those authors 
who have dealt with the interview in the reference process and finally 
zero in on the specific aspects of the negotiation interview and user 
satisfaction fyiat are the focal points of this study. 

If we remember that the use of online bibliographic retrieval in 
libraries is but a decade old, we will not be surprised that the litera- 
ture in the field is relatively recent and though growing at a rapid 
rate, as yet not overly abundant. Moreover, though much of it is des- 
criptive, very little is theoretical or empirically based. Perhaps an 
indication of the growing maturity of the field is the recent appearance 
of several textbooks that synthesize much of the journal literature. 
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The Online Search Process 

A brief overview of the nature of the online search process may be 

had from the following descriptions. 

Seated at a computer terminal, the online searcher 
ts in direct contact, vta a telecommunication link, 
with a remotely located computer system. There, 
machine-readable files called data bases are stored. 
These data bases are essentially electronic versions 
of the indexing and abstracting services familiar to 
librarians. Through a structured protocol established 
by the retrieval system and using subject descriptors, 
key words, and basic Boolean logic, the searcher can 
examine the contents of the data bases. 

(Atherton and Christian, 1977, p. 2) 

And, 

An online bibliographic database search involves the 
direct communication between a searcher and the 
computer system using a conversational program dialogue. 
The searcher, using a specific set of predefined 
commands directs the computer in searching machine- 
readable indexes for certain information. The searcher, 
by typing appropriate commands into a terminal, can tap 
the resources of the entire system. 

(Chan, 1981 , p. 2) 

A personalized view of the actual online interaction is the following: 

You dial a telephone number, not of the search computer, 
but of one of the network's small computers in or near 
your city. Treating this computer somewhat as the 
telephone operator at your friend's office, you "tell" 
it whom you want to "talk" to—all in code. The network 
computer then connects you to the appropriate computer, 
to which you now identify yourself. You do this by 
giving a password, rather than your name. The password 
identifies you as a valid user and implies authorization 
for the search service to bill you for the computer time 
you are about to use. You will also be billed for the 
use of the network through your search service. The net- 
work in other words, treats the search service as its 
customer, and the search service resells network services 
to the consumer. . . 

When your terminal ts finally connected to the computer 
you want to use (networks usually serve many different 
computer services; connection to the network is not the 
same as connection to the computer) and you are identified 
as a valid user, your overhead functions are completed. 
The actual search begins with you telling the computer 
what file you want to search. Then, you are ready to 
begin searching. 

(Meadows and Cochrane, p. 13) 

7 , 
. „, 



These overviews are, of course, succinct summaries of a rather 
lengthy and complex series of actions that require a host of skills, 
knowledge, and Behaviors on the part of the search analyst. Hoover 
(198Q, p. 197-210) feels that before the planning of a search can be 
undertaken, the analyst must first have a baste understanding of the 
equipment, concepts, systems, and data bases to be used. He treats 
such topics as terminals, telecommunications links, search logic, and 
the search system before describing elements of interactive communi- 
cation like system commands, hits or postings, bibliographic records, 
user aids, organization of a typical data base, controlled and un- 
controlled vocabulary, inverted indexing, and data base file structure. 
Only after these subjects have been introduced and explained in some 
detail does he feel the searcher has a sufficient background to approach 
the planning and performing of the search itself. The planning aspect 
of the online search is all the more important if one is concerned with 
the costs of online connect time— costs that may either be passed on to 
the user as search fees or that may be covered by the host organization. 
Before logging on to the search system, he suggests that several steps 
be followed: 

Defining the search topic (the reference interview) 

Determining the type of search needed 

Selecting data bases and search terms 

Creating a preliminary search strategy (p. 210) 
Having followed these steps fairly rigorously, the analyst may now turn 
to the performance of the search and the recording and evaluating of 
the results. While this specific article deals with the mechanics of 
searching, it is worth noting that it comprises only one chapter in a 
guide to online services that presents material on other related areas 
such as producers and vendors of bibliographic online services, promo- 
tion, management, measurement and evaluation of online services, the 
training of searchers, and online user groups, among others. 

Other recent treatments of the online search process also give an 
accurate indication of the complexity of the topic and the various kinds 
of background knowledge that a searcher must have to execute successful 
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searches. Having acquainted the novtce with an overview of the develop- 
ment, growth, and types of databases and online services, Chen launches 
tnto the fundamentals of online database searching (1981, p. 13-60). 
She describes the hardware required including different types of 
terminals and their unique features and proceeds on to the "basics of 
searching 11 briefly explaining the search command options of the DIALOG 
system and the use of Boolean operators. The commands and operators 
are illustrated by excerpts of sample searches. The issue of question 
negotiation is regarded to be of such crucial importance that a separate 
chapter is devoted to it. 

Fenichel and Hogan also separate the chapter of their text devoted 
to the "nitty-gritty" of performing an online computer search from that 
describing the negotiation interview which they call "the reference 
process online style" (1981, p. 67). Although they state that it is 
their intention to convey to the reader an understanding of the search 
process rather than to' teach the reader to perform searches, they too 
consider that the presentation of sample searches will best illustrate 
the concepts they are discussing. Their treatment of the search process 
covers the statement of the search request, the identification of key 
concepts, translating the concepts into terms that match the index 
language of the database, the steps involved in logging on, entering 
the search terms online, combining terms using Boolean logic, printing 
references and logging off. 

Meadow and Cochrane follow the same general pattern in that they 
separate out their discussion of the presearch interview from their 
treatment of the other elements of interactive searching. These elements 
include connecting the user to the data base, the selection of appro- 
priate search terms or phrases, the use of combinational logic, browsing 
and printing. Comments on variations in command languages and the cost 
of online searching conclude this part of their treatment of the basics 
of online searching (1981, p. 9-24). 

Wtlks (1982) presents the most recent, comprehensive treatment of 
wfiat every searcher should know about online searching. His emphasis 



ts not ,so much on the search process as a series of discrete identifiable 
steps, actions, or behaviors, but on the detailed explanation of ten 
specific online interactive commands that he groups into ten functions 
for 

1. logging on and off a particular system 

2. housekeeping 

3. data file selection 

4. browsing the dictionary indexes containing all 
the searchable terms of a database 

5. searching the database itself 

6. logical combining of word sets created to 
refine a search 

7. document or citation display, online or-offHne" 

8. saving search strategies, including editing of 
those strategies 

9. online document ordering, and 

10. carrying out special tasks on a particular system— 
the special features (p. 37). 

He, too, treats the reference interview, as he refers to it, separately 

though somewhat briefly following it with a set of seven sample searches 

illustrating different search strategies, databases, and systems. His 

is a particularly noteworthy addition to the literature because of the 

use of Canadian examples whenever possible. 

From this review of the treatment of the search process in recent 
standard works, a number of things become apparent. First, a set of 
standard definitions of the search process does not yet exist. Does 
the process start when the user submits a search request, when the 
analyst begins to develop a search strategy, when the searcher logs on 
to the terminal, or at some other point? 

Second, there is no standard, universally used terminology to 
describe the stages of the search process. One source lists 61 "quasi- 
equivalent 11 terms to describe 6 concepts in online bibliographic 
retrieval (Meadows and Cochrance, 1981, p. 5)1 -While there does appear 
to be a move toward more standardized usage, the existing diversity 
still creates problems in comparing different studies and in retrieving 
relevant information. 
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Third, the treatment accorded online searching by the recently 
published texts tends to vary in degree of specificity and emphasis. 
While some concentrate on very specific aspects of the search so as to 
constitute virtual procedural manuals, others are more general, 
providing an overview for the informed observer. Regardless of the 
approach taken, however, each author recognizes the importance of the 
negotiation or online reference interview to the search process. A 
more detailed examination of the negotiation interview in online 
searching will be undertaken later in this review. But, as has been 
implied already, the task of assessing user information needs did not 
arise with the advent of computer technology but stems from a long and 
well -documented scholarly tradition centering around the reference 
interview. It is to that literature that we now turn for insight into 
the elements that contribute to a successful interview. 

The Traditional Reference Interview 
The reference process involves "interaction between the librarian, 
the library patron, and the library's resources in order to satisfy the 
patron's information needs" (Jahoda and Braunagel , 1980, p. 1). One 
model of the process describes it as a series of decision steps that 
include the analysis of the user's request so that the subject matter 
and type of information needed are identified; the refinement of the 
query for a more complete understanding; the selection of types of 
answer-providing tools; the location of the answer within a specific 
title selected; the communication of the answer to the user; the asses- 
sing of the adequacy of the answer to meet the user's needs and the 
possible renegotiation of the query if the results are not deemed to 
be satisfactory by the user (p. 2-3). 1 While this model of the 
reference process is one of several that might have been chosen, its 
importance lies in showing the primary place of the reference interview 
in determining the success of the steps that follow. It is the first 
step of the reference transaction and provides the opportunity to 
discover the inquirer's true information need even'if the initial 
expression of that need is not made apparent. From the earlier 
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examination of what went on across the reference desk by Eleanor Woodruff 
in 1897, it was recognized that finding out precisely what the user 
really wanted to know presented a major difficulty (Rothstein, 1977, 
p. 392). Though calls for patience, approachabil ity, and common sense 
dot the -older references, it was not until the last two decades that 
the reference interview was subjected to rigorous analysis. 

William Katz defines the reference interview as "a dialogue between 
someone in need of information and someone—the librarian—able to give 
assistance in finding it" (1982, p. 41). This implies that some accurate 
communication must take place between the individual with the need and 
the one with the means of meeting that need. It is this negotiation of 
the reference question that has been described as "one of the most 
complex acts of human communication" for "in this act, one person tries 
to describe for another person not something he knows, but rather some- 
thing he does not know"(Taylor , 1968, p. 180). This approach focusses 
on the user's difficulty in being able to express' his or her need. Taylor 
suggests that the inquirer has four levels of information need that must 
be worked through: first, the visceral need when there is an actual 
but unexpressed need for information; second, the conscious need when 
there is a conscious, wi thin-brain description of the need; third, the 
formalized need when there is a formal statement of the need; last, the 
compromised need when the question is presented to the information 
system (p. 182). It is the skill of the librarian to work with the 
inquirer back from the compromised to the formalized and possibly even 
to the conscious need and then to develop an appropriate search strategy. 
Taylor found that experienced information specialists pass reference 
requests through five "filters" from which they extract data that will 
aid in the search. These "filters" for structuring the interview are: 

1. determination of subject; 

2. objective and motivation (of the inquirer); 

3. personal characteristics of inquirer; 

4. relationship of inquiry description to file 
organization; 

5. anticipated or acceptable answers (p. 183). 



Each of these "filters" presented problems that could impede the suc- 
cessful outcome of the search-. 

Other scholars have alerted the librarian to other problematic areas 
such as the ambiguity or incompleteness of the query statement, the 
failure to specify the amount of information needed, the failure to 
specify the level of difficulty of the possible answer, the constraints 
imposed By language, time period, place, or type of publication needed 
(Jahoda and Braunagel , 1980, p. 116-123). 

During the past ten years, attempts to come to grips with the 
difficulties posed by the reference interview led scholars to the areas 
of verbal and non-verbal communication for explanatory paradigms. Body 
language or kinesics such as posture, facial expression, hand gestures, 
eye movement, and head nods were shown to influence the reference inter- 
view. For example, facial expression was said to be of importance in 
indicating "approachabil i ty" . Leaning forward instead of away from a 
user indicated interest. BoUcher (1976, p. 31) summarized the two 
extremes that librarians might adopt as the "preoccupation mode" and 
the "availability mode". The former characterized by arms across the 
chest, no eye contact, busy hands, bent head sent negative communication 
signals to the user. The availability mode, on the other hand, evinced 
by relaxed hands, a smile, willingness to leave the desk, head up, eyes 
ready to make contact sent the user positive indications that the 
librarian was prepared to be approached and helpful. Somewhat along 
the same lines, Gothberg (1976) showed that the immediacy of the libra- 
rian's non-verbal communication will positively affect the level of 
satisfaction of the user with the reference interview. Kazlauskas 
(1976) confirmed that the positive non-verbal behavior exhibited by 
librarians such as the use of the eyebrow flash to indicate immediate 
acknowledgement of the user, the use of nodding to indicate that the 
request is understood, the use of eye contact and slight smiling all 
generated similar positive responses from the user whereas negative 
behaviors inhibited user requests and interaction. Munoz (1977) also 
devotes attention to the significance of non-verbal communication in 
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tfie reference interview mentioning physical distance, orientation, 

physical appearance, posture, head nods, facial expression, gestures, 

looking, and paralinguistics. She concludes by stating that the 

reference librarian has 

an obligation to become acquainted with the 
current research in nonverbal communication 
...Nonverbal communication is not just a. 
matter of common sense; the study of it is 
a scientific enterprise which shows us how 
to use the various communication modes as 
conscious techniques. It offers us a means 
to establish more effective contact between 
inquirer and librarian (p. 223). 

The recognition of the importance of empathy on the part of the 
librarian led at least two authors to see a strong relationship between 
the reference interview and counselling and therapy interviews (Peck, 
1975; Pierce, 1971). Others (Shosid, 1974; Horn, 1974) turned to the 
sociological concepts of role and status relationships to point out 
that conditions in which neither the librarian nor the user know what 
to expect from each other, where there is a lack of control on the part 
of the librarian and therefore a need to establish relative status, 
where the user is defensive about having to seek help and uncertain 
about the librarian's ability to provide assistance, i.e., typical 
elements of a reference encounter, all these present barriers to effec- 
tive role relationships and reference transactions. 

The search for explanatory theory though somewhat abated still 
continues. Gavryck (1982, p. 36) suggests' that an exploration of the 
literature on task groups, group interaction, social penetration theory, 
and uncertainty reduction theory would help to further illuminate the 
relationship generated between the librarian and the user during the 
reference interview and that such insight would better enable the 
librarian to create a more cooperative atmosphere that would facilitate 
the flow of information so crucial to a successful search. 

While the behaviors and roles of the participants in the reference 
interview have received considerable attention from some, others have 
concentrated on the techniques of structuring the reference interview. 
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MethodQlogists in the social sciences pointed out the importance of the 
interview as a research instrument and in their efforts to Increase 
respondent participation, minimize situational determinants, and reduce 
and control error they identified those characteristics of the question- 
answer process that would yield reliable, unbiased, valid responses. 
The "openness" versus "closedness" of the questions and evidence of 
encouragement, silences, guggles, and interruptions between questions 
were identified as especially important (Hymari, 1975; Richardson, 
Dohrenwend, and Klein, 1965). "Open" questions, those prefaced by 
"what, when, how, who, where, why" occur most frequently at the begin- 
ning of an interview and encourage the respondent to answer at .length. 
"Closed" questions begin with words like "is, do, can, will" and call 
for shorter, responses . King (1 972) stresses the need for librarians 
to be good interviewers and advocates greater use of the "open" question 
to elicit better responses from the user. White (1981) carries the 
discussion of these types of questions further in her analysis of the 
dimensions of the reference interview. She suggests that the use of 
"open" and "closed" questions affects the'pace, that is, the speed and 
efficiency, of the interchange between question and response ancf that 
the arrangement or sequencing of the questions may occur in three ways: 
moving from open to closed questions (the funnel " sequence) ; moving from 
closed to open questions (the reversed-funnel sequence); and using h 
series of either open or closed questions (the tunnel sequence). The 
choice will depend on the amount of information the user has. about the 
topic and the user's degree of articulateness . 

From this summary, it becomes apparent that the investigation of 
the reference process has a long and rich tradition. This; tradition 
has fostered a reasonably clear and generally accepted view of what 
occurs during the reference interview though why and how it occurs are 
still matters of interest to investigators. In recent years not only 
the findings of other disciplines have provided lenses through which 
to examine the reference interview but their research methods have been 
used to study the phenomenon with greater accuracy and objectivity. 
Terminology is not a major issue in this relatively mature field and 
descriptive models that have been constructed to show the flow of 
events in the reference process are generally accepted. 
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The Online Negotiation Interview 

While a general overview of the computer-based search process .was 
presented earlier, it remains for this section to examine the online 
interview itself in somewhat greater detail . First, the online 
interview will be placed in the context of the broader search process, 
then specific topics such as negotiation behavior, terminal techniques, 
and satisfaction measures will be treated . 

Recent models of the online search process clearly show the place 
of the search interview. Briggs (1976), while describing the entire 
set of interactions between the user and the analyst as the "user 
interface' 1 creates a model consisting of two major components, pre- 
search activities represented in the top part of the* diagram, and 
post-search activities represented in the bottom half. These components 
involve interactive processes that reflected existing user interfaces 
as they occurred and were documented at real search centres. 

Insert Figure 1 Here 

A somewhat simpler model was developed by Fenichel and Hogan 
(1981, p. 68) which described the steps of the online search process 
in a linear fashion as follows: 

Insert Figure 2 Here 

Cochrane's (1981) depiction of the process involves the pre-search 
interview, pre-search staff preparation, search strategy planning, 
online searching, offline post-hoc analysis and possibly even requestor 
use of information obtained. 

All of these models stress the importance of the online negotiation 
interview. Occurring, in the early stages of the search process, it is 
the step that allows the search analyst to determine exactly what the 
user's information needs are and allows the user to learn what the 
computer can and cannot do for him/her. During this interview the 
search analyst must negotiate the information request and transform it 
into appropriate search statements that can be handled by the online 
system and file. Objectives, choice of terms, search strategy must 
all be discussed and mutual agreement reached. Meadows and Cochrane 
lend to our understanding of the presearch interview and the online 
search by depicting them as a series of steps that occur though not in 
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a fixed order (1981, p. 128). These stepk are as follows.: (1) Clari- 
fying and negotiating the Information need\and search objectives. It 
ts now that it is determined whether high recall (retrieving all relevant 
items), high precision (retrieving only relevant items.) or retrieving 
some relevant items is most impprtant to the user and constraints are 
identified. (2) Identifying relevant online system and data bases. 
(3) Formulating basix: search, logic and planning search strategies. The 
search topic is analyzed into concepts and plan£ made for combining 
these. (4) Compiling the search terms. Thesauri and free-text terms 
and their sequence are decided upon. (5) Making putput choices. Limits 
are placed on the ultimate form of the printout. (6) Conceptualizing 
the search as input to the retrieval system. Search terms are arranged 
into concepts using the features (truncation, word proximity) of the 
system, concept groups are arranged in order of importance and sequence, 
output is restricted based on search objectives. (7) Evaluating pre- 
liminary results. Search results are reviewed, alternative strategies 
considered with possible recycling of steps 1-6. (8) Evaluating final 
results. The user's satisfaction with the results are determined. 

In her step-by-step guide to the presearch interview, Somerville 
(1982) divides the components of the interview into four groups: those 
common to all interviews, additional components if thte user is unfamiliar 
with online searching, additional components if the user is present at 
the terminal during the search, components that can be omitted for 
frequent users. Those elements common to all interviews are: (1) the 
use of interpersonal communication and negotiation skills; (2) the 
discussion of the subject with user; (3) the determination whether a 
computer search is the appropriate way to answer the question; (4) 
making sure the search analyst understands the question; (5) determining 
comprehensiveness of the search question; (6) identifying limits; (7) 
selecting data base(s) or system; (8) identifying additional sources; 
(9) identifying main concepts and developing search strategy; (10) 
identifying potential problems; (11) determining alternative strategies; 
02) determining citation output; (13) discussing confidentiality; (14) 
conducting the post-search review. For inexperienced uSers, information 
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on the benefits and limitations of computer searching, software features, 
and potential databases would be added. For repeat users, the Interview 
would be briefer with shorter explanations and discussion of online 
organization, procedures, and search strategy. Among those factors 
that affect the interview, Somerville lists the location of the user, 
whether the user is at the terminal during the search, the searcher's 
knowledge of the subject, the subject speci ficity, and the searcher's 
knowledge of the data base. 

Other researchers have chosen to study the online interview using 
more behavioral approaches. Hitchingham (1979) charts user and searcher 
interaction patterns using the Bales Interaction Process Analysis scheme 
for observing social and emotional behavior of individuals in small 
groups. The profiles she constructed were composed of categories 
representing positive social-emotional areas ("shows solidarity", "shows 
tension release", "agrees"), neutral task areas .("gives suggestion", 
"gives opinion", "gives orientation", "asks for orientation", "asks for 
opinion", "asks for suggestion"), and negative social-emotional areas 
("disagrees", "shows tension", "shows antagonism"). Not surprisingly, 
perhaps, she found that the information-giving activity by searchers 
is most striking and that searchers showed higher levels of giving 
suggestions and opinions while users predominated in the agreement 
category. 

Fidel (1981), using the case study method, constructed a pattern 
model of searching styles which she labelled operational ist and con- 
ceptual ist. According to her, operational ist searchers seemed to base 
their search formulations and interactions mainly on the manipulative 
power provided by the systems. Operational ists 

"understand" a request by translating it into 
search statements. They use the thesaurus as 
the main source for clarification and frequently 
as the only one. They usually look for terms 
submitted by the user; after they find the 
appropriate descriptors and know the category to 
which they belong, they are ready to formulate 
the search. If they cannot find a descriptor to 
represent the concept, they do a free-text search 
of the original terms (p. 69). 
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Conceptualist searchers seem to formulate and modify their searchers 

mainly by performing conceptual analyses. Conceptual ists 

"understand" a request by fitting it tnto.a 
conceptual structure (e.g., faceted structure). 
In order to identify the different elements of 
the structure they may need information that 
was not provided By the user, and they use all 
the sources available to them (e.g., professional 
dictionaries) to acquire this information. They 
persistently search for descriptors to be in- 
corporated into the initial search formulation; 
they may use broader or related descriptors to 
represent an element in the conceptual structure 
if a specific one cannot be identified. During 
the formualtion process they decide which elements 
of the conceptual structure are most important for 
the specific request, and they investigate these 
thoroughly. Most often the preparation continues 
after logging on, and some planning is performed 
online (p. 68-69). 

Fenichel (1980) examined the relationship between searching be- 
havior and searcher experience with the ERIC database. Her results 
showed that novices performed surprisingly well though moderately 
experienced searchers with ERIC experience performed the briefest, 
most cost-effective searches, (measured as time per citation retrieved). 
She found that there was enormous individual variability in searching 
behavior and startling simplicity in a large portion of experienced 
subjects 1 searches. Once formulated, the initial search strategy 
was not modified in half the searches, and only the most basic 
techniques of selecting and combining terms were used. Low recall 
often resulted because obvious synonyms were not used. 

In her study of the behavior of search analysts in presearch 
interviews, Cochrane (1981) itemized all those tasks that other 
researchers had reported took place during the presearch interview. 
The resulting typology of tasks fell into six areas: 

1. Descriptive and tutorial tasks 

2. Request clarification tasks 

3. Request negotiation tasks 

4. Vocabulary construction tasks 

5. Search strategy tasks 

6. Other activities (administrative, diversion, etc.) v 

(P. 4) 
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Specific tasks were listed under each area and the typology was used 
as the checklist to encode and analyze data from videotapes and audio- 
tapes collected of presearch interviews. "Request clarification" was 
shown to occur most often as the first event of the interview while 
"other activities" was most often the last. There was no statistically 
significant relationship between either the librarian or user satis- 
faction scores and the occurrence of the professional tasks listed above. 

Elsewhere Cochrane (nee Atherton) presents a checklist of possible 
social interaction behaviors that may be exhibited by both the search 
analyst and the user (Atherton and Christian, 1977, p. 53; Atherton 
and Jensen, n.d., p. 14). It may be seen from Table 1 that the 
positive and negative occurrences that she lists have their roots in 

Insert Table 1 Here 
the literature of interpersonal communication that formed the explana- 
tory framework for a substantial portion of the work on the traditional 
reference interview. Further parallels between the role of the 
librarian in manual and online searching have been drawn by Knapp 
(1978), Hammer (1981/82), and others concerned with the interview 
process as interpersonal transaction. 

In reviewing these treatments of the online interview, it becomes 
obvious that the interview may sometimes include activities that take 
place at the terminal, especially if the user is present when the search 
is entered. It is also true that measuring satisfaction often depends 
upon the appropriateness of the search output which is a direct result 
of the search techniques of the analyst. Therefore, this review will 
now focus briefly upon recent conceptualizations of the search strategy, 
Detailed discussions of this topic appear in such journals as Online , 
Database , Online Review ; in recent texts (Chen and Schweizer, 1981; 
Fenichel and Hogan, 1981; Jahoda and Braunagel , 1980; Katz, 1982; 
Meadows and Cochrane, 1 981 ) . For further references , readers are 
referred to recerrt bibliographies (Hall, 1977; Hall and Dewe, 1980; 
Hawkins, 1982) and reviews of the literature (Bates, 1981; Bellardo, 
1981; Fenichel, 1980^81; McCarn, 1978) and volumes of the Annual Review 
of Information Science and Technology, 
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Discussions of "search strategy" vary in comprehensiveness and 
specificity. For Chen and Schweizer (1981), the development of the 
search strategy occurs once the interview is completed and involves 
first and foremost the Identification of concepts. The search analyst 
would develop the search strategy as follows: (1) Define the search 
by determining the scope and type of information needed by the user. 
(2) Define the main subjects of the search. (3) List the search terms 
to be used. (4) Develop the search profile using selected terms and 
Boolean operators. (5) Conduct the search. (6) Review the search 
results. (7) Revise the search. (8) Print citation online or offline 
(p. 64). For Meadow and Cochrane (1981), search strategy is also 
defined as "a concept that several steps and levels of work in online 
bibliographic retrieval" (p. 133). They define a search strategy as 
a series of decision points that the analyst must make before going 
online, while online, after the information need has been clarified, 
negotiated, and search objectives established. Decisions to be made 
before going online include determining the order in which concepts 
will be searched online; choosing terms and their forms to represent 
the concepts; deciding on system features to satisfy search objectives. 
While online the analyst must decide how to react to unfavorable pre- 
liminary results and how to revise the search logic for improved results, 

Bates (1981) prefers a narrower definition of "search strategy." 
She uses the term "search mechanics" to describe the operations or 
methods for accessing a data base or search service including means of 
logging on and off, use of Boolean logic and commands. "Search formu- 
lation" she uses to apply to the search statements that express the 
search topic of a request— what many searchers have referred to as 
their "search strategy" for a specific topic,. "Search profiles" 
describe the subject interests of an SDI subscriber; Finally, "search 
strategy" to Bates means "an approach to or plan for a whole search. 
A search strategy is used to inform or to determine specific search 
formulations decisions; it 'operates at a level above term dhoice and 
command use." (p, 142) It is this latter definition that will be 
applied here. 
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tftlks sees search strategy development as an inverted pyramid 
"with broad concepts or subjects of interest listed first and the 
more specific modifying concepts, narrowing the search to produce 
one specific concept" (1982, p. 139). Figure 3 helow illustrates 
this notion. 

Insert Figure 3 Here 

Perhaps the most quoted models of the types of search strategy 
development are those of Charles Bourne and his associates. They 
classified search strategies into five types with the first type used 
most often. Using the '"Building Block" (Figure 4) approach the search 

Insert Figures 4, 5, 6, 7 Here 
analyst divides each search into its component concepts, carries out a 
minisearch for each concept, and combines the result of each minisearch 
to obtain a final result. 

In the "Successive Fractions" approach, the search analyst begins 
by retrieving a large set of a general topic and then intersects 
specific concepts with the large set until it has been reduced to the 
desired size (Figure 5). 

Employing the "Citation Pearl Growing" approach, the search 
analyst begins with a known relevant citation, reviews the index terms 
that were used to retrieve that citation (the "pearl"), then recycles 
these terms as search statements to retrieve more references that are 
similarly appropriate (Figure 6). 

The "Most Specific Facet First" approach involves beginning a 
multi-concept search with the concept that is most specific and 
ending the search if the first set retrieved is satisfactory (Figure 7). 

. The "Lowest Postings First" is similar to the previous approach 
in that it involves beginning the search with the concept that has the 
fewest postings and terminating the search if the initial retrieved 
set is acceptable (Figure 7). 

Hawkins (1982) discusses the pros and cons of eafch approach and 
he suggests one of his own that he calls "interactive scanning". The 
search analyst begins by using abroad concept that will retrieve a 



large set. Then thesauri and other user aids are consulted and the 
appropriate terms and codes are entered. to obtain another large set. 
This wealth of information is scanned for subject information and 
alternative strategies are developed and entered as are additional 
synonyms until a final result is* obtained. He recommends this 
approach as being effective for high recall searches. 

At a more specific level, Bates (July 1979) introduces the 
concept of a search tactic or "move made to further a search". She 
distinguishes between four types of tactics: those that keep the 
search on track (monitoring tactics); those that facilitate movement 
through file structure to the desired file or information source 
(file structure tactics); those that help in constructing the search 
formulation (search formulation tactics); and those that help in 
selecting and revising specific terms that comprise the search 
formulation (term tactics) and gives examples of each. She also 
presents "idea tactics" that are meant to generate new ideas or 
solutions to search problems by improving the search analysts 1 
creative processes (September 1979). 

As one moves from abstract conceptualizations and models to the 
actual performance of the search at the terminal, the literature of 
practical advice to the search analyst becomes far more abundant. 
Examples might include journal articles on broadening or narrowing 
search strategy (Warden, 1977); checking search results to improve 
searching (Marshall, 1980); editing and formatting search printouts 
to achieve a more readily understandable product (Huleatt, 1979), 
and many, many more. 

Measuring Satisfaction 
Of course the ultimate objective of the online search is to fill 
the information needs of the users so that they are satisfied with the 
result. That seemingly simple statement gives rise to a host of issues 
The questions of what to measure, how, when, and for what purpose come 
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to mind immediately. Earlier evaluation studies measured the nature 
and volume of online use—who used the service, which, data bases were 
used, how many Searches were performed. Data bases themselves .were 
the subject of much study with F. W. Lancaster contributing perhaps 
the most used variables to estimate effective information retrieval, 
precision and recall. Because charges were inherent in online 
searching, cost studies that tried to determine such things as the 
average cost per search, per citation, per data base abounded. 
According to Hitchingham (1977), some of the variables included in 
efforts to determine costs by investigators like Elman (1975), 
Benenfeld (1975), Wanger (1975), Cooper and de Wath (1976), and 
Elcheson (1978) included computer connect time, terminal costs, 
communication costs, off-line prints, overhead, and salaries. There 
is still no general agreement either on which of these items should 
6e used to determine cost or what proportion of the costs incurred 
should be passed on to the user. While commercial information brokers 
may charge at a rate that ensures a profit, and university libraries 
may charge for direct costs generated by the search, public libraries 
are reluctant to pass on full costs to patrons, who have been tradi- 
tionally served free of charge. Studies by Summit and Firschein 
(1976) and Lynch (1982) comprehensively document the financial aspects, 
of online searching in public library settings. 

But these administrative aspects of online usage were not the 
only areas that were measured. If the time online was to be measured 
in terms of dollars and cents, then it was obvious that searcher 
efficiency was an important element. So data was sought on what 
constituted an efficient search, the influence of searcher education, 
background, experience, familiarity with data bases .and systems used, 
subject expertise, decision-making ability, personal traits, 
environment factors and so on. 

Not only did what to measure present problems, but how such 
measurement was to be achieved proved equally difficult. Since most 
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Of those involved in providing the service were practicing librarians 
with full work loads, the amount of information to be collected about 
the provision of the search service had to be carefully controlled so 
as not to overwhelm staff-wtth added paperwork. Understandably, the 
information deemed to be most important was that providing statistics 
for administrative purposes. Though such information was valuable it 
was highly dependent on local conditions and did not meet the more 
rigorous requirements of data collected for scientifically designed 
studies. Mick and his colleagues (1980) lament the lack of usable 
user studies. Instruments to collect information might include any 
combination of the following: online search logs, user request forms, 
appointment books or calendars, reference interview records, vendors 1 
invoices, monthly statistical sheets, videotapes, audiotapes, observa- 
tion, user evaluation forms, surveys, and questionnaires. Needless 
to say, there was no uniformity among the instruments themselves. Nor 
was there any way to insure that users would either return their forms 
or fill them out accurately if they did. Hitchingham (1977) has 
suggested four categories of questions to be included on user feedback 
forms: user characteristics questions such as how the user learned 
about the service, what other sources were consulted; performance 
questions relating to precision and recall, comparison with manual 
searches, interview techniques, turnaround time; value statements 
about the overall evaluation of the service; and questions about format 
and support features. Hoover (1977), Fosdick (1977), and Tagliacozzo 
(1977), provide sample user evaluation questionnaires and the findings 
of their research on user satisfaction. Kobelski and Trumbone (1978) 
report that the overwhelming percentage of users in the studies they 
analyzed reported that they were satisfied with their search, a finding 
which raises questions not only about the level expectations and 
sophistication of current users but about the discrimination power of 
instruments used to measure their satisfaction. 



As Tessier and her associates (1977) pointed out these studies 
took, as their focus the user's satisfaction with the search output . 
Such concentration for these researchers represented an oversimplifi- 
cation and they postulated that at least four aspects of computer- 
based retrieval situations contribute to user satisfaction: the output, 
the interaction with the intermediary, the service policies, and the 
library as a whole. Further, they viewed satisfaction as a state of 
mind experienced (or not) by the user. It encompassed both intellectual 
and emotional aspects and is affected by the expectations which 
the user brings to the search process, how well the output of the search 
fits the user's preconceived requirements, and whether the solution 
falls within an acceptable range. In her subsequent work Tessier (1981) 
concentrated on satisfaction measures in presearch interviews since it 
was at this stage of the search that behaviors and tasks would be per- 
formed by the search analyst that would affect the ultimate success of 
the search. Not surprisngly, she found few discussions or scales for 
measuring subjective interview success. From those she did identify 
by Carmon (1975) and Briggs (1976), Hftchingham (1979), Gothberg (1974) 
and Hecht (1979) she concluded that measurement of interview success 
required a set of questions rather than a single question, that summing 
the responses for all questions for a given person improves the reli- 
ability of the measure, that interview satisfaction is multi-dimensional 
being comprised of both task and spcio-emotional components, that 
interview satisfaction is a rather insensitive dependent variable. 

From the literature then, the conclusion was 
drawn to develop a multiple question question- 
naire. The questions, taken together, should 
be varied in content to tap the dimensions of 
satisfaction identified in the literature, Also, 
the questions from previous studies should be 
used as far as possible, in order to allow cross 
study comparisons. The issue of insensi tivi ty 
would be addressed by multiple questions, scale 
construction to enhance variablity, and admin- 
istration procedures, but not by quota sampling 
(1981, p. 4). 



On the whole, however, her study is more interesting for her identi- 
fication of interview dimensions than for the strength of her findings. 
There was a lack of significant relationship between situation variables 
and user satisfaction while the pattern of correlations to librarian 
satisfaction show cpncern by librarians for issues such as charging, 
use of forms, and presence of. the user at'the search. Librarian satis- 
faction was not affected by user characteristics or query characteristics 

In another part of the same large study, Genova (1981) returned to 
the concerns of earlier researchers who had studied the traditional 
reference interview nonverbal behaviors— in this case focussing on 
postural shifts, facial cues such as laughter, eye contact and glances, 
and hand and head movements. The findings are largely inconclusive and 
relate behavior to user satisfaction with the interview but not the 
final outcome of the search, 

Cone! us ion 

This review began by suggesting that a study of the online 
negotiation process and its relationship to the ultimate satisfaction 
of the user grows out of a set of topics, which though closely related, 
nevertheless, have their own research emphases and literatures. It was 
pointed out that the online search process in libraries is barely a 
decade old and that our understanding of it is still expanding and 
deepening. Definitions are plentiful though sometimes contradictory, 
empirical studies are increasing but difficult to compare, method of 
data collection and analysis are widely divergent and too often 
dependent on local conditions to be readily general izabl e. 

The negotiation interview was shown to stem from a long scholarly 
tradition that examined the reference interview in manual searching. 
The theoretical and methodological frameworks that formed the bases of 
these studies stemmed from the social and behavioral sciences especially 
those dealing with communication theory, group interaction, verbal and 
nonverbal behavior. While early studies of online services tended to 
concentrate on financial and other administrative aspect of" the- service, 
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more recent investigations have again returned to some of these areas 
for clues to the ingredients "of user satisfa<J:ion. Research that has 
studied user satisfaction with the output of the search, and that 
which" has focussed attention on the elements of the presearch interview 
are both available. It remained for this study to ltnk some of , the 
areas that have been recognized as being important by concentrating on 
specific behaviors exhibited by the search analyst (pauses, open and 
closed questions), and their relationship with the satisfaction of the 
user using techniques of the social and behavioral sciences that have 
been .shown to be effective. 
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NOTES 

Other models of the reference process appear in Bunge, Charles A. 
Reference service in the information network. In Joseph Becker 
( Ed . ) , Conference on tnteHtfrrary communt cat ions and information 
networks, Airlie House, 1970 . Proceedings. Chicago: American 
Library Association, 1971, 109-116. Also Jahoda, Gerald, and 
Olson, Paul E. Analyzing the reference process, RQ, 1972, 1_2» 
148-156. Katz, William A. rntroduction to reference work . 
Vol. II. (3rd ed.). Toronto: McGraw-Hill, p. 108. 

Based on the earlier unpublished work of C. P. Bowne, B. Anderson, 
and J. Robinson, descriptions of these models are given in 
Buntrock (1979, p. 12-13), Fenichel and Hogan (1981, p. 71-74), 
Hawkins and Wagers (1982, p. 18-19, the source of the Figures 4-7 
above), and Meadow and Cochrane (1981.*. p. 136-139). 
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Figure 1. Model of Existing User Interface 

Source: Briggs, R. Bruce. The user interface for 
bibliographic search services. Proceedin gs of the 
Clinic on Library Applications of Data Processing, 
University of Illinois , 1975: The Use o f Computers 
in Libraries , University *f Illinois, Graduate 
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Figure 2. Overview of the Online Search Process 

Source: Eenichel, Carol Hansen, and Hogan, Thomas H, 
Online searching: a primer , Marlton, N.J.: 
Learned Information, 1981, p. 68. 
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know abou t online searching . Ottawa: Canadian 
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Figure 7 
Most Specific or Least Posted Facet First 
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Source: Hawkins, Donald T. , and Wagers, Robert 

Online bibliographic search strategy development. 
Online , May 1982, p. 10-19. 
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TABLE 1 



CHECKLIST FOR OBSERVING INTERPERSONAL COMMUNICATION 
DURING P RE-SEARCH INTERVIEW 



INFORMATION SPECIALIST 



* USER 



Positive Occurrences' 



Initially, used open questions 

*Encouraged discussion 
Answered questions in under- 
standable way 
Thoughtful pauses before 
answering 

Summarized or paraphrased 
request 

Listened to user 
*Gave full attention 
Remained objective about 
subject of request 

Appeared comfortable and 
relaxed 



Freely stated information need 



Asked questions freely 



Appeared confident in skill 
of information specialist 

Listened to information specialist 
*Gave full attention 



Appeared comfortable and t^laxed 



Negative Occurences 



9 
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Initially, used closed 
questions 



interrupted or talked-over 
often 

Gave command or directives. 

expecting compliance 
Attempted to demonstrate 

superior 
*Placed user on defensive 

*Gave erratic attention 

Reacted subjectively to 
request 

Exhibited uneasiness 

Appeared competitive 
Appeared submissive 
Ended interview prematurely 
Seemed annoyed 



Had to be prompted to give 
information 

Changed topic often 

Showed indecision about choices 

*Frequently interrupted or talked- 
over 

Objected to suggested strategies 

Exhibited insufficient knowledge 
about subject 
*Placed information specialist on 
defensive 



Exhibited uneasiness 

Appeared competitive 
Appeared submissive 
Terminated interview prematurely 
Seemed annoyed 



Source: Atherton, Pauline, and Jensen, Becky, 
I nterfaces in computer-based bibliographic 
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ABSTRACT 



Previous research has suggested that in the 
interaction between the user with his/her 
knowledge arid understanding of his/her in- 
formation needs, and the search analyst, 
with his/her understanding of the retrieval 
system and data bases, lies the key to 
understanding why some searches are more 
successful than others. In our research, 
we have conducted a series of experiments 
in order to determine the relationship* be- 
tween interview techniques, knowledge gain, 
and us6r satisfaction with the references 
retrieved. This paper is a preliminary 
exploration'of the effects of two interview 
techniques: the donscious use of ''open 11 
and '•closed 11 questions and the use of 
pauses of different lengths i>y the search 
analyst during the online negotiation 
? nterview. C? 
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INTRODUCTION 

The development of online bibliographic search services, which 
provide the possibility of conducting automated library searches, is one 
of the most important revolutions which has occurred in library science 
in the past decade (Wanger, et aK , 1976). These automated search ser- 
vices provide an alternative to the labor-intensive process of manually 
locating bibliographic citations in indexes such as Psychological 
Abstracts , ERIC , and Social Sciences Citatlpn Index . But, unlike other 
innovations in library automation that have effected mainly technical 
serv j ces (e.g., computerized cataloguing, circulation systems, etc.), 
online bibl iographic services affect the relationship existing between 
the library user and the library materials. Instead of direct access 
to the indexes, the user of an online bibliographic service typically 
has intermediaries between him/herself and the materials: a search 
analyst (usually a specially trained reference librarian), and the 
computer system itself. The analyst must comprehend the search request 
and devise an appropriate search strategy, keeping in mind the needs of 
the user, the capabilities of the system, and the constraints of the 
service. The system largely determines the type of information that may 
ultimately be retrieved, the limits within which the search may be made, 
the format of the bibliography retrieved and printed, and the cost of 
the search. 

In this paper, we are concerned with the first of these interme- 
diaries, namely^the search analyst, with special attention given to how 
various interview techniques used by the analyst can affect search re- 
sults and user satisfaction. Obviously, inadequate communication be- 
tween the user and the search analyst can result in the development of 
an inadequate search strategy which fails to retrieve bibliographic 
citations relevant to the user's concern (Atherton and Christian, 1 977) - 
Hence, the ability of the search analyst to elicit pertinent information 
from the user and to convert this information into an appropriate search 
strategy must be viewed as an extremely important skill. 

METHOD 

In the research reported here, we undertook a series of experi- 
ments that looked closely at the role and behaviour of the search analyst 
in order to determine the relationship between the interview techniques 
that were utilized and the user's level of satisfaction with the quality 
0 f references retrieved. To do so, we obtained the assistance of two 
volunteer search analysts at different Canadian university libraries who 
expressed an interest in participating in the study. They underwent a 
one-day training session to become familiar with the interview techniques 
that we wished to explore before beginning the control led experimental 
interviews. 

Because of the difficulty of trying to employ several techniques 
in the same interview, the search analysts were asked to experiment with 
one technique at a time. Specific techniques to be investigated included 
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"open" vs. 'Closed 11 questions, and the use of pauses of different lengths 
to stimulate the client to elucidate his information need. Open ques- 
tions refer to those open-ended questions which require more than a few 
words for an adequate response, whereas closed questions can be ade- 
quately answered in a few words (identification of person, place, number, 
etc., or a yes-no response.). Pauses refer to those periods during which 
the interviewer waits without speaking for the user to begin speaking or 
continue speaking. Both question type and length of pause have been 
identified by Richardson (1965) as being related to the quality of inter- 
viewee response, and were also noted in exploratory studies of negotia- 
tion interviews undertaken by Auster and Lawton ( 1 979) • 

Open questions were expected to stimulate responses which had 
greater breadth ancj depth than those for closed questions. It was 
believed that these extensive responses to the open questions would tend 
to provide a more valid map of the user's problem area than would 
responses to the closed (identification, selection, yes-no) questions. 
In other research, it was found that the use of pauses in the interview 
tended to have an effect on the user's participation in the interview 
process, with shorter pauses (under 10 seconds) stimulating deeper^ 
participation by the user, and longer pauses (over 10 seconds) inhibiting 
greater participation (Richardson, 1965). We expected the same results 
In the research being reported here. 

As a control for each of the experiments, a series of 15 'naive' 
interviews were conducted by each analyst before they were trained in the 
four techniques (i.e., open question, closed question, short pause and 
long pause). Fifteen interviews were then carried out for each of the 
two levels of the open/closed, and short pause/long pause variables. 
Hence, a total of 75 interviews were conducted by each of the two 
analysts, yielding a sample size of 150. 

The order in which the interviews of each of the four types were 
done was chosen in a random fashion, so as to ensure that any systematic 
trends that might develop were removed from the treatment effects. The 
naive .control interviews were conducted prior to any training or exposure 
to the four experimental techniques, since it was believed that any such 
training would likely alter the search analysts 1 behaviour. 

Users were screened so that only those seeking retrospective sub- 
ject matter searches in the social sciences were included. Any given 
user was included only once in the study. It was expected that other 
user traits (experienced or inexperienced; student or professional, etc.) 
would be randomly distributed among the sample. We recognize that these 
traits may in fact not be randomly distributed, but there is no practical 
mechanism for controlling these variables experimentally. 

All search interviews were recorded on audiotape and later- 
analyzed to ensure that the proper experimental technique had beerr\ 
administered. Quantitative measures of the number of open and closed 
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questions and mean pause time were employed for use in the data analysis 
phase of the research. Follow-up questionnaires were also conducted 
after the user had received his/her computer print-out of bibliographic 
citations. These questionnaires were designed to determine the extent 
to which the user was satisfied with eight different aspects of the 
search process, including the helpfulness of the search analyst, the 
usefulness of the bibliography, the currency of the search results, 
and the cost. All eight factors were then combined to form an overall 
index of user satisfaction that was also used in the data analysis phase 
of the study. 

Statistical analyses consisted primarily of two one-way multi- 
variate analysis of variance with blocking on the search analyst. 
Stated another way, we used a two-way analysis of variance where the 
second factor was "search analyst 11 . In that way, variation in the 
dependent variable (user satisfaction) due to differences between the 
two search analysts was removed. Since there were 30 subjects per cell 
for each of the levels of the experimental variable, it was felt that any 
appreciable effects of the experimental variables would be statistically 
significant at the 0.05 level. 

Statistically, the null hypothesis to be tested in each case was 
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where y is the control group mean on the dependent variable i, y_ is the 

group mean on the dependent variable for the first experimental condition 
(e.g., open question or short pause) and y g is the group mean on the 

second experimental condition (e.g., closed question or long pause). 

At the data analysis stage, we first checked to see how accurate 
our measure of user satisfaction was, and also how well our search 
analysts completed their interviews in terms of the proper use of pauses 
and open and closed questions. With regard to our eight-item scale of 
user satisfaction, we obtained a Hoyt estimate of reliability equal to 
0.79, indicating that our index of satisfaction was quite satisfactory 
for bur intended purposes. 

To determine the success of the search analysts in carrying out 
the various experimental interview techniques we employed analysis of 
variance, using mean pause time and proportion of questions open as the 
dependent variables, and analyst and experimental technique as the 
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independent variables. These analyses showed that analyst B, a more 
experienced search analyst, administered the open and closed question 
experiments more successfully, while analyst A administered the pause 
experiments more successfully. As was expected, in the case of both 
analysts, a higher proportion of interview questions were open, when the 
interviewers were originally classified as open, longer pauses were 
evidenced when the interview was classified as "long pause 11 , and shorter 
pauses were evidenced when the interview was classified as "short pause 11 . 
This suggested to us that, for the most part, the four required experi- 
mental techniques were properly administered by the two search analysts, 
although the actual proportion of open questions asked by the two 
analysts were less than originally anticipated, and the overall mean 
pause time shorter than originally expected. 



FINDINGS 

As you will recall, the major focus of attention in this paper is 
the relationship between the four experimental research techniques em- 
ployed by the two search analysts and the level of user satisfaction 
reported by the user. The results of this analysis are reported in 
Tables 1 and 2 presented below. Table 1 shows the effects of the experi- 
mental sessions using open and closed questions on user satisfaction 
level. Here we see that out of a total score of 100, the overall level 
of satisfaction was found to equal 76.88, with a score of 81.^5 for those 
searches resulting from sessions using open questions and a score of 
for those searches resulting from sessions using closed questions. It 
should also be noted that the level of satisfaction in the "control" group 
was lowest of all, with a mean score of 7^.73. 

TABLE 1 

THE EFFECTS OF OPEN AND CLOSED QUESTIONS 
ON USER SATISFACTION 



Mean 
Satisfaction 

Level 
(Total=100) 



Standard 
Deviation 





CONTROL 

Analyst A 
Analyst B 
Mean 


70.78 
78. AO 

74.73 


13.78 
14.23 
14.28 


13 
14 

27 




OPEN QUESTION 
Analyst A 
Analyst B 
Mean 


78.90 
83.75 
81 .45 


11.53 
9.88 
10.68 


9 

10 
19 


• 


CLOSED QUESTION 
Analyst A 
Analyst B 
Mean 


76.43 
74.80 
75.40 


14.63 
• 10.03 
11.55 


7 
12 
19 




OVERALL MEAN 


76.88 


12.70 


65 
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TABLE 2 

THE EFFECTS OF SHORT AND LONG PAUSES 
ON USER SATISFACTION 



Mean 
Satisfaction 

Level 
(Total=100) 



CONTROL 

Analyst A 
Analyst B 
Mean 



70.78 
78.40 
74.73 



LONG PAUSE 

Analyst A 
Analyst B 
Mean 



70.50 
70.83 
70.73 



SHORT PAUSE 

Analyst A 
Analyst B 
Mean 



76.68 
75.68 
76.13 



OVERALL MEAN 



74.25 



Standard 
Deviat ion 



13-78 
14.23 
14.28 



13 
14 

27 



15-95 
13-52 
13.83 



5 
9 
14 



10.23 
12.80 

11.43 



9 
11 

20 



13.25 



61 



These findings are in agreement with our model outlined above. In other 
words, it was found that sessions using more open-ended questions in 
which the user was required to give more detailed and lengthy responses 
to the analysts' queries resulted in a higher level of satisfaction with 
the search results. 

When one is actually involved in the search process in this manner, 
it is not surprising that the user feels that she/he has provided as much 
information as possible for the analyst, so that the analyst can recover 
relevant citations from the computer system. 

A closer examination of Table 1 also shows that in the case of the 
open question experiments, the level of satisfaction for Analyst B (83-75) 
was higher than that expressed for Analyst A (78.90). In the case of the 
closed question sessions, however, the reverse is true, with a higher 
level of satisfaction evidence for Analyst A (76.43) than for Analyst B 
(74 80) A possible conclusion that we can. make here is that Analyst B 
may'have used open-ended questions more effectively than Analyst A for 
eliciting information, while Analyst A may have employed closed questions 
in a more advantageous manner than Analyst B. However, forboth analysts, 
the total level of satisfaction expressed by the user was higher when 
open questions were being employed. However, it should be noted that the 
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standard deviations are relatively large in Table I, showing a consider- 
able deviation from the mean in the case of both open and closed questions. 

Table 2 shows the effect of short and long pauses on overall user 
satisfaction. Here, we see that the overall level of satisfaction is 
74.25 out of 100, with user satisfaction being higher when short pauses 
were employed (76.13) than when longer pauses were employed (70.73). 
The control sessions yielded an overall satisfaction level of 7**«73» in 
between the other scores. These findings are also in agreement with the 
model that we outlined above. You will remember that we hypothesized 
that user satisfaction would be higher when shorter pauses were employed 
and that lower satisfaction would be evidenced when longer pauses were 
employed, since shorter pauses were expected to cause the user to volun- 
teer more information, whereas pauses of longer duration would cause a 
feeling of awkwardness and discomfort in the user. 

The difference in levels of satTsfaction expressed for each 
analyst was very slight in the course of the pause experiments, with 
Analyst A receiving a slightly higher score for short pauses (76.68 as 
opposed to 75.68 for Analyst B) , and Analyst B receiving a higher score 
for long pause sessions (70.83 as opposed to 70.50 for Analyst A). 
Table 2 shows relatively large standard deviations from the mean in the 
case of both long and short pauses, and this should be considered when 
interpreting the mean scores given in the table. 

IMPLICATIONS AND CONCLUSION 

While these findings reflect only a preliminary analysis of the 
data collected over the two year period of this study, they do suggest 
some directions to be considered in the training of online search 
analysts in negotiation techniques related to the areas we examined. 
With regard to the use of open questions in the interview process, there 
seems to be a reluctance on the part of analysts to employ this mode, 
even after they have been urged to do so. Since there seems to be a 
significant relationship between user satisfaction and the use of open 
questions, it would seem reasonable to suggest that training in the use 
of this technique would be advisable to increase user satisfaction. 
Further, since a one-day training session increased the use of this mode 
only moderately, It would appear that more prolonged, concentrated, in- 
tensive training opportunities would be desirable. Methods of training 
might include role playing and analysis of audio tapes and video-tapes of 
interview sessions to help the analyst learn more effective interview- 
bphavior patterns. 

With regard to the long and short pauses, some related and addi- 
tional Implications for training might be drawn. Analysts might be made 
aware of the positive relationship between shorter pauses and ultimate 
user satisfaction. Since there seems to be a natural tendency toward the 
use of shorter pauses, it would appear that intensive training to employ 
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th i s technique is not warranted. However, practice in using this tech- 
nique in conjunction with open questions might prove to be beneficial 
in producing a more satisfied user. 

Finally, we urge readers to exercise caution in interpreting and 
applying these findings since this paper is the first report of the 
preliminary analysis of data. Also, we do not advocate the abandonment 
of traditional training methods but rather a modification of these to 
include techniques that have been found to be positively related to user 
satisfaction. 
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ABSTRACT 



A series of controlled experiments were undertaken to determine 
the relationships among the interview techniques of the search analyst, 
the knowledge gain of the client, and the client's satisfaction. 
Techniques investigated included "open" vs "closed" questions, and the 
use of pauses of different lengths. This paper details the analytical 
techniques developed to examine the relationships. 

INTRODUCTION 

The purpose of our research- was to undertake a systematic investigate 
into the relationships among: (1) the techniques used by search analysts 
during preliminary interviews with clients before engaging in the on-line 
retrieval of bibliographic citations; (2) the amount of new information 
gained by the client as a result of the search; and (3) the client's 
ultimate satisfaction with the quality of items retrieved. In this paper 
we present a detailed discussion of our research methodologies, with major 
attention given to the clarification of the research questions initially 
posed, and the research design implemented in order to provide answers to 
those questions. Research techniques, data collection instruments, and 
statistical analyses employed will all be examined in the following pages, 
in order to provide a comprehensive overview of the research that was 
undertaken. We begin with a discussion of the research questions initially 
posed for the study. 
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RESEARCH QUESTIONS 

In order to gain insight into the relationships among the three 
factors discussed above—the knowledge of the client, the interview 
techniques of the search analyst, and the client's satisfaction with 
the bibliography and materials— we felt it was necessary to formulate 
specific questions that could be applied to a series of controlled 
experimental sessions involvfng the search analyst and the client. Two 
key research questions were finally posed for this purpose. They are 
as follows: 

1. What are the relationships between various interview 
techniques employed by the search analyst and the 
amount of information gained by the client? 

2. What is the relationship between the amount of 
information gained by the client, the value he/she places 
on this information, and his/her satisfaction with the 
bibliography and materials? 

The first question required that we make four assumptions, which 
were to be validated during the study. These assumptions are as follows: 

1. that the client's responses to questions are related to 
the type of questions asked; 

2. that the search strategy developed by the analyst is 
closely related to these responses; 

3. that the citations retrieved are closely related to the 
particular search strategy used; and 

4. that the information gained by the client is related to 
the content of the citations retrieved. 

We believed that these assumptions were reasonable based on the findings of 
Auster and Lawton (1979), and Richardson (1965), which provided considerable 
evidence that interview technique does affect the validity of the response. 

The second question carries one implicit assump&on; namely, that the 
client f s gain in information is proportional to the reduction in his/her 
uncertainty. With this assumption, and the analysis of the value of 
information provided earlier, we inferred that if high value is placed on 
a small information gain, then the client must have been near the break- 
even point , 
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Believing this analysis to be correct, we hypothesized that 
satisfaction with bibliographic materials retrieved would be positively 
related to both the value and amount of information gained, with value 
being more important than amount. This suggested the rank-ordering of 
satisfaction scores as shown in Figure 1. 



Gain 



Value 





High 


Low 


High 


1 


2 


Low 


3 


4 



Figure 1 -- Hypothesired rank ordering of satisfaction 
scores in relation to the value and gain 
in information. 



PROCEDURES 



In order to determine the relationship between the interview 
techniques utilized by a search analyst and the client's level of 
satisfaction with the quality of references retrieved, we decided 
to undertake a series of controlled experiments in which specific 
interview techniques would be used, and their effects analyzed. 
Because of the difficulty of trying to employ several techniques in 
the same interview, it was decided that the search analysts should 
only experiment with one technique at a time. Specific techniques to 
be investigated included "open" vs "closed" questions, and the use of 
pauses of different lengths to stimulate the client to talk. Open 
questions refer to those open-ended questions which require more than 
a few words for an adequate response, whereas closed questions can be 
adequately answered in a few words (identification of person, place, 
number, etc., or a yes-no response). Pauses refer to those periods 
during which the interviewer waits without speaking for the client to 
begin speaking or to continue speaking. 

According to the literature, open questions were expected to 
stimulate responses which had greater breadth and depth than those for 
closed questions. It was believed that these extensive responses to the 
open questions would tend to provide a more valid map of the client s 
problem area than would responses to the closed (identification, selection, 
yes-no) questions. Also, in other research it was found that the use of 
pauses in the interview tended to have an important effect on the client s 
participation in the interview process, with moderate pauses (under 10 seconds) 
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stimulating deeper participation by the client, and no pause 
(under 1 second) inhibiting greater participation (Richardson, 1965, 
p. 204). We expected the saine results in the research being reported 
here . 

As a control for each of the experiments, a series of 15 "naive" 
interviews were to be conducted by each analyst before they were trained in 
the four interview techniques (i.e., open question, closed question, 
short pause, and long pause). Fifteen interviews were then to be carried 
out for each of the two levels of the open/closed and short pause/long 
pause variables. Hence, a total of 75 interviews were planned for each 
of the two search analysts participating in the study making a total 
sample sixe of 150 interviews. 

The order in which the interviews of each of the four types were 
done was chosen in a random fashion so as to ensure that any systematic 
trends that might develop were removed from the treatment effects. The 
naive control interviews were conducted prior to any training or exposure 
to the four experimental techniques, since it was believed that any such 
training would likely alter the search analyst's behaviour. 

Clients were screened so that only those seeking retrospective 
subject-matter searches were included. Any given client was included only 
once in the study. It was expected that other client traits (experienced 
or inexperienced; student or professional, etc.) would be randomly 
distributed among the sample. We recognize that these traits may in fact 
not be randomly distributed, but there is no practical mechanism for 
controlling these variables experimentally. 

All search interviews were recorded on jaudio tape for later analysis 
to validate assumptions and to ensure that the proper experimental technique 
had been administered. Additional information was provided by the search 
analyst on a ore-page questionnaire designed to record technical aspects 
of the search process, such as the purpose of the search, interview time, 
strategy time, systems used, data bases accessed, number of citations 
printed, and so forth (see Appendix A). 

Follow-up telephone questionnaires were also administered after the 
client had received his/her computer print -out of the bibliographic 
citations. If the person could not be contacted by telephone, a copy of 
the questionnaire was mailed to the address given by the client at the 
time of the search. These questionnaires were designed to determine the 
extent to which the client was satisfied with the search process, the 
usefulness of the citations received, the amount of knowledge gained from 
the search and the expertise of the client in the area before the search 
was conducted (see Appendix B). 
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The search analyst was also requested to submit a copy of the 
search strategy entered into the computer system in order to generate 
the desired list of citations. These four pieces of information, (1) 
the audio tape of the search interview, (2) the questionnaire filled 
out by the search analyst regarding the technical aspects of the search, 
(3) a follow-up questionnaire completed by the client concerning higher 
expertise in the subject area and the value and outcome of the search, 
and (4) a copy of the actual search strategy employed, constituted the 
main sources of data with which we tested our two research questions. 

Information contained in the questionnaires completed by the 
search analyst and client were coded and entered directly into the data 
file used in the statistical analyses. The taped interview and search 
strategy, however, required additional analysis before the necessary 
information could be entered into the data file. In the case of the 
taped interviews, you will recall, we were interested in determining 
whether the desired open question, closed question, short pause, and 
long pause experimental techniques had been successfully administered. 
To check this, the entire search interview conducted by the search 
analyst was tapped out 11 on special sheets, which enabled us to record 
the entire format of the interview, including when a question was asked, 
whether the question was open or closed, when the client responded, the 
length of the pauses involved, and so forth. For more information on this 
"mapping" process, see the example given in Figure 2. (A sample of the 
coding sheet is included in Appendix C.) 

The actual content of the search strategy developed by the analyst 
for on-line use was also required for the validation of assumption one, 
associated with the first research question. The nature of the decoding 
that was involved in the analyses of these search strategies is also 
explained in Figure 2. 



The following excerpt is taken from one of the tape- 
recorded interviews, conducted by a search analyst as a 
part of this study. Here, we use the excerpt to demonstrate 
how we analyzed the taped conversation in order to provide 
data for the various phases of the study. This example, 
we hope, will clarify some of the questions that might arise 
in this paper. Each question and response is numbered (1 
through 12) to make it easier to follow the analyses involved. 
The complete search strategy, incidentally t involved eight major 
concepts and 32 steps. 
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Tape 67 



1. Analyst: Have you ever had a computerised search 
done before? 

(0 seconds) (closed question) 



2. Client: 

3. Analyst: 

4. Client: 



Yes. 

(0 seconds) 



(no pause) 



So you know what you're going to get out of 
it; what you're going to get at the end? 
(0 seconds) (closed question) 

Well, What I got before was a series of titles 
with some brief descriptions of content. 
(1 second) (short pause) 



5. Analyst: 



Client: 



7. Analyst: 



8. Clients 



9. Analyst: 



0K--Would you please tell me a little bit about 
your topic? 

(0 seconds) (open question) 



1 2 3 4 5 
OK — The topic is the 
9 10 11 

boards of education 
16 17 18 19 

process — and how the 
25 26 27 28 
fit into the general 
32 33 34 

negotiated. Do the 
39 40 41 42 
and those are formal 
(14 seconds) 



6 7 8 

negotiation process between 
12 13 ~T4 15 
and teachers — the bargaining 
20 21 22 23 24 
goals of the various groups 

30 31 
framework of what's 
35 36 37 38 
goals affect what's negotiated- 
43 44 45 

goals; stated goals, 
(long pause) 



OK. I've never done one (a search) on this, so 
this should be interesting— OK, so should I also 
put the bargaining...? 
(2 seconds) (closed question) 

Yes, there's also the bargaining component. I 
mean teachers negotiate under the bargaining 
act— they qualify to bargain with the board of 
education. 

(1 second) (short pause) 

OK. In here (thesaurus) they have negotiation 
agreements and they have negotiation influences. 
Have you been through ERIC already? 
(0 seconds) (closed question) 



ERIC 



JL f«y JL 



10. Client: Yes. 

(0 seconds) (no pause) 

11. Analyst: Oh good—OK. What one's (descriptors) 

did you find that were most beneficial? 
(0 seconds) . (open question) 

12. Client: I'm not sure... 



Tue following diagram shows how we ''mappefd" each 
taped interview to determine the number of open and 
closed questions, and the number of short and long pauses, 
A map for an entire interview shows every question asl\ed 
and every response given, in addition to the time 
(in seconds) between questions and responses. 



c C* closed 

2. O R ^ 
j : — — 0= open 

3. C 0 _ 
^ R= response 
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In validating assumption one, we required the 
total number of words, seconds and major concepts that 
were contained in responses to open and closed questions. 
Using the first open question in the excerpt as an 
example (i.e., #5), we found that there were 45 words, 
20 seconds, and 4 major concepts (major concepts are 
underlined) in the response. 

In assumption two, we were interested in determining 
whether the major terms and concepts used by the client 
in the description of the topic were actually employed 
by the search analyst in the search strategy. A close 
match between the two would provide validation for our 
assumption. To check on the match, we first listed 
all major terms and concepts used by the client in the 
taped interview, such as the three concepts underlined 
in the excerpt above. Ke then examined the on-line 
search strategy and again listed all major terms and 
concepts. A comparison of the lists provided the 
necessary base for comparison. 

SEARCH STRATEGY EXCERPT : 

1. Arbitration . Collective Bargaining . Contracts . 

LahnT Demands 

2. Board of Education 

3. Secondary School Teachers , Teachers 

4. Goals , Objectives 

5. 2 and 4 

6. 3 and 4 

7. -1 and 5 

8. 1 and 6 

9. 7 or 8 
10. Print 9 
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Terms Used by the 

Client During the Interview 



Terms Used by the 

Search Analyst in the Strategy 



negotiation process 
•boards ---education 
•teachers 



arbitration 
•collective bargaining 

contracts 

labor demands 
•boards of education 

secondary school teachers 
•teachers 
•goals 

objectives 



•bargaining process 
•goals 



6> 



•common terms, used by both client and analyst. 

For this second assumption, we also constructed a 
five-part scale designed to measure the complexity of the 
search strategy employed by the analyst on-line. The 
five items in the scale are listed below, along with the 
resulting figures derived from analysis of the sample 
search strategy. By adding the five items together, a 
total measure of search strategy complexity was calculated. 

SEARCH STRATEGY COMPLEXITY: 

^ l. Number of terms used « 9 

(all terms are underlined in strategy) 

2. Number of operations ("and" "or" and "and not") * 5 
(all operations are circled) \ 



3. Number of major concepts * 4 
(e.g., search statements 1 to 5) 

4. Number of steps in the strategy * 10 

5. Number of data bases accessed « 1 
Total score: 29 




Figure 2: 



Sample of Interview Transcript, 
Search Strategy and Coding Procedures 
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ANALYSIS OF RESEARCH QUESTION ONE : 

1. What are the relationships between various 
interview techniques employed by the search 
analyst and the amount of information gained 
by the client? 

In question one, we attempted to look at the effects of four 
interview techniques— open question, closed question, short pause, 
and long pause — on the amount of information gained by the client. 
Statistical analysis consisted primarily of two-way analyses of 
variance with blocking on search analyst. In that way, variation 
in the dependent variable (e.g., information gain) due to differences 
between the two search analysts could be controlled for. 

Since there were 30 subjects per cell for each of the levels of 
the experimental variable, it was felt that any appreciable effects 
of the experimental variables would be statistically significant at 
the 0.10 level. Statistically, the null hypothesis to be tested in 
each case was: . 



V A c "A "A. 

where A is the control group mean on given dependent variable, 

is the group mean on the dependent variable for the first experimental 
condition (e.g., open question), and^U^ is the group mean on the 

second experimental condition (e.g., closed question). 

Before we began the analysis of our data we first checked to 
see how reliable our information gain scale was. This was accomplished 
by calculating a Hoyt estimate of reliability, using the computerized 
LERTAP program (Nelson, 1974). The reliability of the index formed by 
summing items 12 to 16 (with scoring for item 12 reversed) was 0.50. 
Scores ranged from 7 to 23 (of a possible 4 to 24); the standard error 
of measurement was 2.42 (see Appendix B) . 

As was indicated earlier, four assumptions associated with the 
first research ^question were also to be validated in the course of the 
study. In the following paragraphs these four assumptions will be 
re-stated along with a description of how we attempted to provide 
the necessary validation. 
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Assumption one stated that the client's responses to questions 
asked by the search analyst would be related to the type of question 
asked. As was indicated earlier, we expected longer, more complex 
responses to follow open questions and moderate pauses, but it was 
uncertain as to how strong this relationship might be. To validate 
this assumption, we decided first to randomly choose 25 open and 
25 closed questions in 50 tapes randomly selected from all of the 
taped interviews conducted by each search analyst. This would result 
in "double randomization" of our sample questions; first by tape, 
and second by question within tape. For each of these 100 questions 
(50 per analyst), we then proceeded to determine the number of words 
in the response to the given open or closed question, the number of 
seconds in the response, and the number of major concepts included in 
the response. Obviously, if responses having more words and concepts, 
and being of longer duration followed open, as opposed to closed, 
questions, then our assumption would be validated for type of question 
asked . 

To test the same assumption for short and long pauses, we chose 
ten cases of long pause and ten cases of short pause from all searches 
carried out by the two search analysts, and again determined the 
numbers of words, seconds, and concepts in the response following the 
given pause interval. Here, we took the ten most extreme cases for 
each analyst; that is, the five cases with the shortest pauses and the 
five cases with the longest pauses. No randomization in selection was 
undertaken for pause responses, since we wanted to see how clients 
responded when pauses were of moderate length, or absent. Our assumption 
would be validated if the more complex responses followed moderate 
pauses. 

The secdnd assumption stated that the search strategy developed 
by the analyst) would be closely related to the responses given by the 
client. In ot)her words, it was expected that the search analyst 
would employ cbncepts and phrases used by the client in the interview 
process in thej actual search strategy being created for the on-line 
bibliographic retrieval service. To validate this assumption, we 
proceeded to ciarry out ten case studies (5 per analyst) in which we 
carefully examined the relationship between the vocabulary used by 
the client and the discrete terms and concepts employed by the analyst 
in the final search strategy. Five tapes Were randomly selected from 
each analyst and all key terms and concepts used by the client in 
his/her description of the topic being searched were recorded. 

Hie final version of the search strategy drawn up by the search 
analyst and used in the on-line search process was then examined, 
and all key terms and concepts were again recorded. By comparing 
the two lists of terms and concepts, we attempted to discover how 
closely the search strategy corresponded to the description of the 
search topic that was given by the client (see Figure 2 for more 
information) . 
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For this second assumption, we also decided to relate the 
length of the entire interview to the complexity of the search 
strategy employed, with the belief that more complex search 
strategies would result from search interviews that required 
mpre discussion time on the part of the analyst and client. 
Complexity of search strategy was based on a five-part index 
consisting of the following items: (1) the number of key 
terms used in the search strategy, (2) the number of operations 
(e.g., "and" "or" and tf not" employed to join the key terms 
together into concepts) , (3) the total number of major concepts 
included in the strategy, (4) the total number of steps in the on-line 
search strategy, and (5) the total number of data bases finally 
searched. The rationale behind this index was a belief that more 
complex bibliographic searches consist of a larger number of terms 
and concepts that are searched on several different data bases. By 
correlating the length of time in the total interview with this five- 
part index, we hoped to uncover another possible source of validation for 
our second assumption. (More information on the validation of this 
assumption is provided in Figure 2.) 

Assumption three stated that the citations retrieved would be 
closely related to the particular search strategy used. As a rule, one 
might expect more complex search strategies to produce bibliographies 
with higher percentages of hits. Since we could not control for the 
subject matter searched, and since only one search was conducted for 
any given request, we concluded that it was impossible to satisfactorily 
validate this item within the design of the present study. 

Assumption four stated that the information gained by the client 
would be related to the content of the citations retrieved. This 
assumption was previously validated in an earlier study (Lawton, Auster 
and To, 1979). Here, clients were asked, "How much did you learn 
about your topic as a result of the search? (1) Nothing or very 
little, (2) Some, (3) A great deal" and "How much of the relevant 
information retrieved was new to you? (1) 0%, (2) 1-10% (3) 10-20% 
(4) 21-50% (5) more l;han 50%." These questions measure perceived 
information gain directly. The fact the path coefficient connecting 
response to the first of these two items to the client's reported 
"satisfaction with the value of the bibliography itself" and "the 
materials located via the bibliography" (coded (1) low, (2) medium, 
(3) high ) was .67, provided construct validation for that it*n. The 
second item was highly intercorrelated with the first. While we considered 
trying to measure information gain from the bibliography directly, we 
judged such an approach to be impracticable. It would require developing 
a series of subject matter tests on different topics for administration 
before and after a search. 
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ANALYSIS OF RESEARCH QUESTION TWO : 

2. What is the relationship between the amount of information 
gained by the client/ the value he/she places on this 
information, and his/her satisfaction with the bibliography 
and materials? 

In question two, you may recall, we hypothesized that satisfaction 
with bibliographic materials would be positively related to both the value 
and amount of information gained, with value being more important than 
amount . 

The first step, in the analysis of data was to determine the 
reliability of the various scales. In several cases, on the basis of 
item analysis, items included in the questionnaire were omitted from the 
scale used to analyse the data in Order to improve scale reliability. 

The various scales and their characteristics are as follows: 



Value of information sought : items 25-32 
on User Response form (App. B) • 

Reliability .69 Standard Error 2.06 

Maximum possible 30 Observed 24 
Minimum possible 6 Observed 6 

Satisfaction with products of search : items 18, 20, 21, 
227 23 on User Response form. 

Reliability .86 Standard Error 1.30 

Maximum possible 25 Observed 25 
Minimum possible 5 8 

Having validated the scales, the next step is to carry out the 
analysis of data. It is planned to answer question two by using multiple 
linear regression with either continuous or dichotomous measurement of key 
variables. Figure 1 suggests the use of dichotomous variables, one for 
each of the four cells, so that the hypothesized additive effects of the 
value and amount of information gain can be estimated. However, an analysis 
using the continuous variables in this form may also be carried out. 



CONCLUSION 

As the foregoing implies, analysis of the data continues with a 
completion date anticipated by the end of the summer. We are hopeful 
that our work will not only provide some answers to questions that have 
proved inordinately peiplexing in the past but will contribute a set of 
validated items toward to development of a measuring procedure that 
has proved to be so elusive in the past. 



9 

ERLC 
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PART V 



ERLC 



OUTLINE OF PROJECT 
PREPARED FOR SEARCH ANALYST TRAINING SESSION ' 

I . Purpose 

A. Two questions to be answered 

1. What are the relationships between various interview techniques 
and the amount of information gained by the client? 

2. What are the relationships between the amount of information 
gained by the client, the value he/she places on this information, 
and his/her satisfaction with the bibliography and materials? 

B. Strategy to answer questions 

1. Identify key concepts/variables ^ 

2. Measure the variables 

3. Relate the variables \ 

C. Key concepts 

1. Interviewer techniques 

2. Information: before, after, and difference (gain) 

3. Value of information 

4. Satisfaction with results 

D. Other concepts 

1 . User traits 

2. Search analyst traits 

3. Search content 

4. Data base/system 

I I . Interviewer techniques 

A, Approaches to measurement 

1. Naturalistic observation 

2 . Exper i menta I cont ro I 
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B. Findings of natural istie research 

1. Types of questions vary In terms of the type of response the 
quest ionner expects* 

a. Closed questions 

i Identification; e.g., What is your sex? 

ii Multiple choice; e.g., Are you male or female? 

iii Yes-no; e.g., Are you male? 

b. Open 

i Questions that require more expansive answers; 
e.g., Teli me about yourself. 

2. Pace of questionnlng differs 

a. Rapid fire — no time to think 

b. Slow — time to reflect 

3. Questions may incorporate assumptions made by the questionner 

a. Leading question — When did you fire the gun at her? 
Assumptions: that you fired the gun at her . 

<D © 

4. Many others 

C. Controlled interview experiments 

1. Purpose: to establish cause-effect links between type of 
interviewer technique and interviewer response. 

2. Types of experiments 

a. Laboratory experiments 

I Artificial setting maximum control over nuisance 
variables 

ii May not be valid in real life 

b. Field experiments 

i * Low control over other variables 

ii More likely to be valid in real settings 

3. Experimental variables used. 

a. Closed-open 

b. Pauses*: short and lortg 

D? Design of this study 

1. Two field experiments 

2. Open-closed j 

3. Short pause vs 10 second; pause 13 ( 
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III. User Sample 

A. Who Is in the study? 

1. Clients of oocial sciences (esp. education) search services. 

2. Retrospective topic searches only, 

a. No "current awareness" where topic has already been defined. 

b. No special purpose searches--e.g. r publication date. on 
a book. 

3. Each searcher is in once and only once. 

4. All searches negotiated (If not run) in person. 

5. Other posslb le' traits are being ignored in the selection process, 
but may be important, and traits will be noted; e.g., 

a. Sex 

b. Educational level 

c. Position, etc. 

d. System and data base used 

e. Topic searched 

IV. Instruments for Data Collection 

A. All data refer to one of the following: 

1. Search analyst— characteristics, behaviour, etc. 

2. User 

3. Search process (e.g., length, strategy, etc.) 

4. Bibliography (e.g., no. of items) 

B. Sources of data 

1. Tapes recording of interviews • 

2. Copy of search strategy, from printout (preferred) or personal 
records 

3. Search Analysts Record of Online Search 

4. User Responses to Online S earch Request 

5. Biography or vita of search analysts 
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Measurement of variables 



EXperir 



i menta I var i ab I es r ^ 

■4 

a. Open-closed - research officer taped verification f may use 

b. Pauses - research officer taped verification \ "' Tanscr iP"'" s ' 

K may sample by 

c. We are looking for M on the average" differences V^tlme 
of substantial magnitude but not absolute perfection. 

We are not sure of exactly what is feasible, but 
suggest that for the subject area negotiation, 

i Open = 70/£ open questions, 3055 closed 
Closed = 70/f closed; 30% open 

li Short pause = 2 sec. ^2 sec. (0 to 4 sec.) 

Pause = 10 sec. + 3 sec. (7 to 13 sec.) for 95/6 of all. 

2* Interviewing variables 

a. Search strategy - analysis' of complexity by research officer - 
no. of terms, no. of conjunctions, etc. 

b. Search characteristics - provided by interviewer 

i Interview time (SA 8) 

II Strategy time (SA 9) > etc. 

3. *' Information 

r Before - User items #5-11. Scores could be added to 
form an index which would provide a more reliable 
measure than any single item. 

il Gain - User Items #12-16. Can form indsx. 

4. * Value of information 

i User #25-32. Can form index. 

5. * Satisfaction with search results 

i User #17-24. Can form index 

6. All items have been taken from 

i Our previous research 

ii Previous research of others 

ill Developed to fit the concepts we want to measure — 
reliability increases with the number of items. 



Questions were later renumbered. 
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Conduct of experiments , 

A. Make sure treatment Is given 

I Validity check to be made by research officer 



B. 



Control "nuisance" variables, If possible - nuisance variables are 
those that may be related to the variables being studied and that 
interfere with the experiment. E.g.: 



1 



Subjects too heterogeneous - reason why identical white rats 
are used rather than ten different types of animals. 
Solution: screen subjects. 



2. 



Trend in data -'e.g., if all "open" interviews were conducted 
first, and during that period most subjects were students, and 
if closed interviews we conducted next, and during that period 
most users were professionals, the difference between results 
might appear to be caused by question differences, when in 
fact it was due to different types of clients. 

C. Solutions r to controlling nuisance variables 

1. Statistical - e.g., compare females with females, males with 
males; students with students; etc. Too difficult a method 
in practice. 

2. Randomization - mix it up so that effect of any trends will 
disappear. That's our choice. Interview treatments have 
been randomized in order shown on sheet Sequence of 
Experimental Interview Techniques: A and B . 

D. Administration of instruments to users - by mail, phone, and person. 
Analysis of Da+a 

A. Basically, compare satisfaction and knowledge gain and other scores 
for 3 groups for each experiment (control, open and closed; or 
control, short pause and long pause) while removing the effects 
due to different interviewers by "substracting" the average 
difference in knowledge gain and other scores across a I I 
i nterv i ews . Graph i ca I I y : 




Control 


Exp. 1 


Exp. 2 
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